To get started, it’s important to understand the state transitions and the associated watcher events inside a ZK client. A ZK client can be in one of the 3 states, disconnected, connected, and closed. When a client is created, it’s in the disconnected state. Once a connection is established, the client is moved to the connected state. If the client loses its connection to a server, it switches back to the disconnected state. If it can’t connect to any server within some time limit, it’s eventually transitioned to the closed state. For each state transition, a state changing event (disconnected, syncconnected and expired) is sent to the client’s watcher. As you will see, those events are critical to the client. Finally, if one performs an operation on ZK when the client is in the disconnected state, a ConnectionLossException (CLE) is thrown back to the caller. More detailed information can be found at the ZK site. A lot of the subtleties when using ZK are to deal with those state changing events.

The first tricky issue is related to CLE. The problem is that when a CLE happens, the requested operation may or may not have taken place on ZK. If the connection was lost before the request reached the server, the operation didn’t take place. On the other hand, it can happen that the request did reach the server and got executed there. However, before the server can send a response back, the connection was lost. If the request is a read or an update, one can just keep retrying until the operation succeeds. It becomes a problem if the request is a create. If you simply retry, you may get a NodeExistsException and it’s not clear whether it’s you or someone else have created the path. What one can do is to set the value of the path to a client specific value during creation. If a NodeExistsException is thrown, read the value back to check who actually created it. One can’t use this approach for sequential paths (a ZK feature that creates a path with a generated sequential id) though. If you retry, a different path will be created. You also can’t check who created the path, since if you get a CLE, you don’t know the name of the path that gets created. For this reason, I think that sequential paths have very limited benefit since it’s very hard to use them correctly.

The second tricky issue is to distinguish between a disconnect and an expired event. The former happens when the ZK client can’t connect to the server. This is because either (1) the ZK server is down, or (2) the ZK server is up, but the ZK client is partitioned from the server or it is in a long GC pause and can’t send the heartbeat in time. In case (1), when the ZK server comes back, the client watcher will get a syncconnected event and everything is back to normal. Surprisingly, in this case, all the ephemeral paths and the watchers are still kept at the server and you don’t have to recreate them. In case (2), when the client finally reconnects to the server, it will get back an expired event. This implies that the server thinks the client is dead and has taken the liberty to delete all the ephemeral paths and watchers created by that client. It’s the responsibility of the client to start a new ZK session and to recreate the ephemeral paths and the watchers.

To deal with the above issues, one has to write additional code that keeps track of the ZK client state, starts a new session when the old one expires, and handles the CLE appropriately. For my application, I find the ZKClient package quite useful. ZKClient is a wrapper of the original ZK client. It maintains the current state of the ZK client, hides the CLE from the caller by retrying the request when the state is transitioned to connected again, and reconnects when necessary. ZKClient has an Apache license and has been used in Katta for quite some time. Even with the help of ZKClient, I still have to handle things like who actually created a path when a NodeExistsException occurs and re-registering after a session expires.

Finally, how do you test your ZK application, especially the various failure scenarios? One can use utilities like “ifconfig down/up” to simulate network partitioning. Todd Lipcon’s Gremlins seems very useful too.

For example, in search engines, for each term t, the index, or called inverted index, contains an inverted list, which is typically a sequence of sorted integer document IDs (and other information which can also be considered as sequences of integers). Thus, inverted index compression techniques are concerned with compressing sequences of sorted integers.

A graph is often implemented as adanjency lists. In many cases, each list can be easily organized as a sorted integer array. For example, for the social graphs in large-scale social networks like Linkedin or Facebook, each list is, for a particular member, a sequence of all his friends (represented as integer member IDs). The performance of many algorithms on such graphs is thus greatly affected by the efficiency of various operations on such lists. For example, in order to find all common friends of two members, we need to find all intersected member IDs of their friend lists.

A matrix can be considered as an alternative implementation of a graph especially when most nodes are directly connected with each other. However, when the matrix is sparse (which is very common for the first or second degree friends in social graphs), it is more efficient to first transfer it into the adancency lists and then do various operations on the resulting lists.

In the above applications (large scale search engines or social networks), we often need to process a huge amount of data (arrays of integers) within milliseconds. The data often need to be compressed to be hold in main memory. Due to compression, the disk traffic and the network traffic are also greatly reduced since much less amount of data need to be communicated. We also need to be able to decompress the data very efficiently to maximize, for example, the query throughput of search engines. To achieve these goals, large search engines have been trying a lot of methods. For example, Lucene uses variable-byte coding (please refer to Managing Gigabytes for various inverted index compression methods) to compress indexes. Google also uses variable-byte coding to encode part of its indexes a long time ago and has switched to other compression methods lately (In my opinion, their new method is a variation of PForDelta which is also implemented in Kamikaze and optimized in Kamikaze version 3.0.0). Therefore, we can see that it is very important to build Kamikaze on top of a good compression method that can achieve both the small compressed size and fast decompression speed.

Kamikaze implements PForDelta compression algorithm (or called P4Delta) which was recently studied and has been shown by paper[1] and paper[2] to be able to achieve the best trade-off of the compression ratio and decompression speed for inverted index of search engines. Many other techniques for inverted index compression have been studied in the literature; see Managing Gigabytes for a survey and paper[2] and paper[3] and for very recent work, especially the detailed performance comparison between most of those techniques and PForDelta. Unfortunately, Lucene does not support PForDelta now although paper[2] and paper[3] have shown that PForDelta can achieve much better performance than variable-byte coding in terms of both compressed size and decompression speed.

Kamikaze builds an platform on top of PForDelta to perform efficient set operations and inverted list compression/decompression. Kamikaze Version 3.0.0 inherits the architecture of the first two versions and supports the same APIs. In Version 3.0.0., the PForDelta algorithm is highly optimized such that the performance of compression/decompression and the corresponding set operations are improved significantly.

In Linkedin, Kamikaze has been used in the distributed graph team and search team. We are also looking forward to contributing to the Lucene community with Kamikaze, especially the optimized PForDelta compression algorithm.

Hedwig
Benjamin Reed (Yahoo! Research)
June 7, 2010

ABSTRACT

Hedwig is a large scale cross data center publish/subscribe service developed at Yahoo! Research. We needed a scalable, fault tolerant, publish/subscribe service that has strong delivery and ordering guarantees to maintain the consistency of replicas of datasets in different data centers. We found that we could use ZooKeeper, a coordination service, and BookKeeper, a distributed write-ahead logging service to build such a service. To present Hedwig I will first review two of the services it is built on: ZooKeeper and BookKeeper. I will then present the motivation for Hedwig, review its design, and present its current status.

BIOGRAPHY

Benjamin Reed has worked for almost 2 decades in the industry: from an intern working on CAD/CAM systems, to shipping and receiving applications in OS/2, AIX, and CICS, to operations, to system admin research and Java frameworks at IBM Almaden Research (11 years), until finally arriving at Yahoo! Research (3 years ago) to work on distributed computing problems. His main interests are large scale processing environments and highly available and scalable systems. Dr. Reed’s research project at IBM grew into OSGI, which is now in application servers, IDEs, cars, and mobile phones. While at Yahoo, he has worked also worked on Pig and ZooKeeper, which are Apache projects for which he is a committer. Benjamin has Ph.D. in Computer Science from the University of California, Santa Cruz.

[This video is part of LinkedIn's tech talk series.]

At scale and with relevance, faceted search makes a lot of sense on the rich structured data we have here at LinkedIn.  The fundamental paradigm was to provide individuals with an easy and natural way to slice and dice through search results or simply content.  We thought that if the content being indexed had some structural dimensions (say like in eCommerce) or could be augmented by implicitly deriving these dimensions with quality, then a Faceted search paradigm would be ideal not only for retrieval but also for Navigation and Discovery.  At Linkedin since a member profile does have these rich structural dimensions, along with rich text data, it seemed that it would be only a matter of time to create such an interface.

At Linkedin we had first developed a prototype exhibiting the traditional pattern; a click on a facet value would be similar to a filtering of search results through that value.  More explicitly, you would search for “John” and later clicked on the “San Francisco” facet value to get only people in San Francisco called John, i.e. “John” + facet_value(”San Francisco”) = “John AND location:(San Francisco)”.  Since a facet value would be displayed only if it contained results, it would never lead you to a dead end while you were navigating through results.

Our next steps were to bring it in front of users through a very nice useability lab (read forthcomming blog).  Feedback was really good with respect to the idea but with respect to facet interactions, users really wanted the ability to select multiple values within a facet.  More explicitly, the ability through facets to look for say “John” in San Francisco or Los Angeles at the same time i.e. “John AND (location:(San Francisco) OR location:(Los Angeles)”.  This with an interface that would naturally provide that behavior and understandable counts.  As you can imagine, we were very excited with the positive feedback.

What we have implemented is essentially a query engine for the following type of query:

SELECT f1,f2…fn FROM members

WHERE c1 AND c2 AND c3..

MATCH (fulltext query, e.g. “java engineer”)

GROUP BY fx,fy,fz…

ORDER BY fa,fb…

LIMIT offset,count

deferring this query to a traditional RDBMS on 10s - 100s millions of rows with sub-second query latency SLA is not feasible. Hence we built a distributed system that handles the above query at internet scale.

The technical challenge is really a caching and counting exercise.  Naively one can view the operation of determining scores for Facet Values as iterating over all search results and tabulating the right Facet Value.  Of course when you have more than a hundred thousand Facet Values spread across tens of Facets, the naive approach doesn’t really scale. With multi-select options counting can also be rather tricky.  More specifically, selected values do not participate in counting their respective facets. A naive solution would be to do N (the number of facets) iterations over the result set for each multi-selectable field. But performance wouldn’t be good and would degrade as we add more such facets. To deal with it, we devised a solution that only iterates the result set once by integrating hit counting and hit validation into a single step.

One item we wanted to highlight in this post is that not all Facets are equal.  If you take a closer look at your results you’d notice that facets can be:

- Static:  A static Facet is ‘Location‘ ; it reflects a static property of the search record and its value is not affect by the searcher neither the search query. In this case it simply counts and tabulates, by location, the number of individuals within the entire search hits.  Furthermore some static facets values are not mutually exclusive.  In the case of ‘previous company’ a member can have multiple values if he had two or more previous employers.

- Dynamic: A dynamic Facet is ‘Recently Joined‘ ; it reflects a static property of the search record but its value is affected by the query.  In the case of the “When Joined” facet, it tracks the difference between the time of the query and when the member joined, rounded to the day.  Note here that we are not re-indexing the joined date on a daily basis.

- Personal: A personal Facet is ‘Relationship‘,  it depends on the distance in the social graph between the member found and the searcher.  We are a social network site hence social distance is a very important facet to provide to our users.  Its nature is actually very dynamic as new connections occur every milliseconds which in turn transform the social network of everyone involved to the 3rd degree.  This makes the social graph so dynamic, that we do not store the relationship between a member and a potential searcher within the index!.

To be good citizens of the community, we have contributed the technology to the open-source world:

• Bobo
• Zoie
• Sensei

What is next?  Fundamentally it is all about speed so next play is to make it even faster and more intelligent

Hadoop drives many of our most powerful features at LinkedIn.  About half of our Hadoop jobs are submitted by Apache Pig.  This means that along with Azkaban and Voldemort, Pig is a large part of LinkedIn’s data cycle - the process behind features like People You May Know and Who Viewed My Profile.

I have used Pig intensively for about a year.  During that time, I have come to love Pig for what it enables me to do: easily manipulate my data at scale, to turn raw data into data products.  As a recovering Perl hacker (see: enlightened perl, catalyst framework), I always employ the tool with the highest level abstraction that fits the job - higher level tools being more powerful.  Because of this, good alternatives to Pig like Cascading, and the exciting work with LISP REPLs in Cascalog don’t do it for me.  If Perl is the duct tape of the internet, and Hadoop is the kernel of the  data center as computer, then Pig is the duct tape of Big Data.  Pig lets me easily flow my data in parallel with simple commands.  It lets me flow my data through dynamic languages like Python if I want to use SciPy, through simple UDFs in Java if I want to use a function repeatedly and share it with others, and ILLUSTRATE lets me check the output of my lengthy batch jobs and their custom functions without having to do a lengthy run of a long pipeline.  Taken together, these features enable me to be productive.

I learned Pig not because I had a big data problem, but because I wanted to build a better interface for Hadoop (see: PigPen, WireIT, this demo video).  For a long time, I did not delve very deeply.  There was no reason to do so: I didn’t have to know how to code in MapReduce - Pig ‘just worked.’  I issue SQLish commands in Pig Latin, and Pig parses these commands and creates and submits MapReduce jobs for me.  This saves me from having to think too hard about the complexity of Java, MapReduce or Hadoop.  I don’t like to think about anything but the problem I’m actually solving, and so while I have written Algebraic MapReduce jobs as Pig UDFs, I am unlikely to ever write a Java Hadoop job unless I absolutely have to.

Apache Pig is now fairly robust, but data-flows themselves can get complex fast.  I’m pretty fluent in Pig Latin, but my code in any language rarely runs on the first try.  With batch computing, running jobs repeatedly to debug them can take a long time and slow development to a crawl.  One must often massage the Pig to command its will.

When I write Pig Latin code beyond a dozen lines, I check it in stages:

• Write Pig Latin in TextMate (Saved in a git repo, otherwise I lose code)
• Paste the code into the Grunt shell - Did it parse?
• DESCRIBE the final output and each complex step - Did it still parse?  Is the schema what I expected?
• ILLUSTRATE the output - Does it still parse?  Is the schema ok?  Is the example data ok?
• SAMPLE/LIMIT/DUMP the output - Does it still parse?  Is the schema ok?  Is the sampled/limited data sane?
• STORE the final output and see if the job completes.
• cat output_dir/part-00000 (followed by a quick ctrl-c to stop the flood) - Is the stored output on HDFS ok?

When you first tackle a complex task with Pig, that last step rarely happens on the first few tries.  In time, you get more proficient.

As an incurious Pig user, I thought of Pig as a black box: a program with a command line.  Nevertheless, I got to know the idiosyncrasies of each version as Pig matured from version 0.2 to 0.7 - unfixed bugs, unusual behaviors, and undocumented limitations.  I never knew exactly why Pig behaved as it did, but I learned to get along with it.

Working on Pig

Several months ago I decided to work on the Pig project.  I don’t even know Java.  I’ve been faking it my entire career (ask me to write a Java class without any Java code around it in an IDE - I can’t do it), so I’m going after low hanging fruit the committers haven’t gotten around to and leaving the tough bits to them.  Log analysis is a common use of Pig, and logs usually contain timestamps, so I want to add a Joda-Time DateTime data type to Pig.

But that is way too hard, so I’m going after boolean first.  I checked out the code.  I worked on it all weekend.  I made a patch.  I made many patches, actually.  Time and again, I thought I was done, but I wasn’t.  Booleans would load in grunt, so I thought it worked - but they wouldn’t store.  I added physical storage code, so I could load and store.  I emailed the LinkedIn Hadoop users list proclaiming victory… but it wouldn’t work on Hadoop.  So I added Hadoop storage code, and it would load and store on Hadoop - but I couldn’t use operators to check for equality.  I added code for ILLUSTRATE and it would illustrate, but I still couldn’t use booleans in a real job.  This went on and on, and the patch remains incomplete (I’ll finish it soon).

During that weekend of long and frustrating hours of Pig hacking, the pattern became familiar.  I was interacting with a different part of Pig each time I got a new kind of error.  The hops from package to package in writing the patch corresponded to the stages of my long hours of stepwise data-flow checks in Grunt, as I had written Pig scripts most days over the course of the last year.

From a user’s perspective using the Grunt shell, this system seems like a cohesive entity - a single program - a complete (and somewhat irrational) Pig.  It doesn’t seem that way anymore.  Now that I’ve read the code, using Grunt is different.  Knowing the way it all fits together at a high level - by tracing exceptions and seeing the package names of classes I’ve failed to implement because I didn’t know they existed or were required - I know that pig is actually segmented into many logical parts, independent arms that verify and process Pig Latin code independently and in different ways.  The interface presented by grunt presents an illusion of wholeness that a deeper understanding of pig makes transparent - clear as illusion.

Complex Systems: Software and the Brain

Watching Pig’s boolean data type’s slow and stepwise recovery reminded me of something else, something personal.  In February, 2009, I had a minor car accident.  I drove home to the farm, and was fine until the next day when I became mute and started blacking out.  If I hadn’t been terrified and crying, the 911 call would have been hilarious to hear played back because it took me several minutes to ask for help.  I had a concussion, and the effects of the injury remained after the concussion passed.  I was punch drunk and irritable for months.  My rate of speech varied from normal to mute, and every range in between.  My brain was throwing exceptions my body could not catch. When I could not talk, I couldn’t think out loud either.  But I could still type.  I could still tweet.  I made a diagram of the way my speech would phase in and out, like a malfunctioning queue that would grow and shrink.  I showed it to my neurologist, but she didn’t pay much attention.  Call it infographic therapy.

At some point I started to consciously observe the malfunctions, often repeatedly in the case of my speech.  The mind is a complex system, and it can fail in parts.  Over time I came to know, at an experiential level, that consciousness is actually an illusion presented to the user: an illusion made up of many independent processes that only appear cohesive when presented as a unified and intuitive interface.  As my brain healed, I got to contrast functional sub-systems with malfunctioning sub-systems.  As a result, I know my limitations better and can apply myself more effectively.

The Data Revolution

For me, understanding my work over the last year by understanding Pig was profound.  It gave it more meaning, because strangely enough Pig has become a big part of my life.  By the numbers, I’ve spent as much time in the last year with Pig as with anything or anyone else in my life excepting my wife.  I’ve never much contributed to open source before, and I’m glad to be transitioning from a passive consumer of other people’s work to an active participant in an open source project.  It is good to create openly, to give back.  Open source is technical righteousness.

But more than that, this is an important time in computer science, and unlike many previous technical revolutions, this one is happening completely in the open.  Like the integrated circuit before it, MapReduce is producing a paradigm shift that opens broad opportunities to produce new kinds of products from our massive collective backlog of data to help people in new and unprecedented ways.  At LinkedIn we’ve amassed the world’s premiere data-set on the labor of professionals, and it is the mission of LinkedIn Analytics to leverage that deeply meaningful data to provide insight and value to our users.  At LinkedIn Analytics data processing is both personal and meaningful, as the features we create enhance the working lives of tens of millions of people.

The Integrated Circuit solved the Tyranny of Numbers and unleashed Moore’s law, enabling a computerized, networked society.  It did so with the considerable overhead of patent licensing and litigation.  MapReduce is solving the Tyranny of Threads, enabling any company to process data at scale in parallel to extract real value from our most abundant and underutilized resource: information.  It is doing it in the open, through free and open-source software, through the Apache Foundation, Hadoop and its sub-projects.  We’ve gotten more efficient organizationally this time around.

One of the reasons I joined LinkedIn Analytics is its commitment to open source.  At LinkedIn, we love open source.  We’re committed to contributing to Hadoop and Pig and giving back to the open source community through projects like Azkaban and Voldemort.  We are determined to provide the open source community with the complete and painless data cycle that we enjoy - to enable even casual hadoop users to analyze data from their application at scale, to mine it for value and store it easily and reliably so that it can drive use and close the data loop.  Look for new open source tools and projects from LinkedIn Analytics in the coming months that will help make this possible!

If you love open source and you love big, meaningful data - we need you.  Come join us.  LinkedIn Analytics is hiring!

( Shout outs to Pete Skomoroch for acting as late-night editor, helping me dramatically improve this post! )

The OS people made POSIX, and mostly adhere to it and it is available on most any system you would use. The language people made java, the clr, common lisp, and a variety of other platforms, none of which really work together.

I like Java. It is a simple, statically compiled language with great performance and solid garbage collector implementations; but unfortunately it was made by the language people. This means a lot of basic functionality available on every platform you could reasonably run on, isn’t available to you when your program is written in java.

Want to move a file across volumes, figure out what user you are, get your process id, send a signal, interact with pipes, etc? Too bad. Sun has decided you shouldn’t, so you can’t.

The motivation for this is to allow java to be used in set-top boxes or something like that, but it presents a bit of a problem for server-side software development when the code needs to get anywhere near the operating system.

You are supposed to be able to fix this with JNI. JNI is a standard that lets you wrap native code with java interfaces and call it from java. The problem is it is a massive pain to write, so people rarely do.

We have seen a number of problems that this causes. We have seen a couple of problems caused by attempts to use Runtime.exec to run a unix command to collect basic file information. This is a major problem since it forks a huge java process to run an itsy bitsy unix command, then attempts to parse the output. Who could be so stupid, you would ask? Well one was buried in an apache commons library, the other was Hadoop (which tries to execute whoami). The later, in addition to forking a large namenode process, is actually quite non-portable in comparison to getuid().

I didn’t know this existed, but it turns out there is a library called jna that does transparent native access for java using dynamic proxies. Here is an example of using it to implement kill() to send a signal to a process in java using jna.

public class Kill {

    public static void main(String[] args) {

        Posix posix = (Posix) Native.loadLibrary("c", Posix.class);
        int pid = Integer.parseInt(args[0]);
        int signal = Integer.parseInt(args[1]);

        posix.kill(pid, signal);
    }

    public interface Posix extends Library {
        public int kill(int pid, int signal);
    }
}

The only downside of this that I have found is that JNI bound native libraries (whether using JNA or not) cannot be reloaded. This means that if you are writing something deployed in a servlet container like tomcat that expects to be able to reload various sub-applications this will not work. The work around is to add the JNA jar to the tomcat system classloader, but this breaks the ability to package the application as a war that works in an container. (Special thanks to a coworker who pointed this out to me before we started converting things right and left.)

Binary search, right?

Wrong. There is actually a method called interpolation search, in which, rather than pessimistically looking in the middle of the array, you use a model of the key distribution to predict the location of the key and look there.

Here is a simple example: assume you have an array of length 10, which contains a uniform sample of numbers between 0 and 99. Interpolation search works the same way a person would search. If asked to guess the location of 3 you would probably guess the 1st slot, if asked to guess the location of 85 you might guess the 9th slot, etc.

Okay, so this approach seems to make better guesses, but does it actually require asymptotically fewer iterations on average?

The answer is that in fact it requires exponentially fewer iterations–it runs in lg lg N time where N is the length of the array. The analysis is a little tricky–it appeared 19 years after the algorithm was formally published (according to the Knuth Search and Sorting book). I had to look it up (and before I could look it up I first had to realize I didn’t invent it and figure out what the name of it was), but essentially each step reduces the range to N^0.5 instead of 0.5 * N which yields the better asymptotic runtime. This is an average case result, so it is worth noting that the variance in the number of comparisons is lg lg N as well. This means that assuming your keys are well distributed you will almost certainly get the average case time or very close to it (I tried it on random arrays and the theory is scarily accurate).

So why isn’t this used in practice? Probably because lg N is already really small. After all, if you have an array of length 2^32 this only drops you from ~32 to ~5 comparisons which in practical terms probably isn’t a big speed up for searching arrays.

But we found a really great use for it in Voldemort. One use case we support is serving really large read-only files as Voldemort stores. This allows us to support a big batch datacycle run out of Hadoop as described here. The data structure for these uses a large sorted index file to do lookups, what is stored in this file is an MD5 of the key. Since the MD5s are used for the sort, the file is guaranteed to be uniformly distributed over the key space, and can often be many GBs in size. These files are memory mapped to help reduce the cost of a read, but the improved search algorithm can help to greatly reduce the number of seeks when the index is not fully memory resident. A disk seek comes at a price of around 10ms, so saving even one or two is a huge performance win.

Sometimes it is nice to see what these things are like in real life.  For uniformly distributed values, given a key to search for, an array to search in, and a minimum and maximum value it might look something like this:

int interpolationSearch(int key, int[] array, int min, int max) {
    int low = 0;
    int high = array.length - 1;
    while(true) {
        if(low > high || key < min || key > max)
            return -1;

        // make a guess of the location
        int guess;
        if(high == low) {
            guess = high;
        } else {
            int size = high - low;
            int offset = (int) (((size - 1) * ((long) key - min)) / (max - min));
            guess = low + offset;
        }

        // maybe we found it?
        if(array[guess] == key)
            return guess;

        // if we didn't find it and we are out of space to look, give up
        if(guess == 0 || guess == array.length - 1)
            return -1;

        // if we guessed to high, guess lower or vice versa
        if(array[guess] > key) {
            high = guess - 1;
            max = array[guess-1];
        } else {
            low = guess + 1;
            min = array[guess + 1];
        }
    }
}

You can see the real deal implementation in Voldemort here–it is a little trickier as it uses non-integer keys and is searching a file but the basic outline is the same.

1. Keynote by Jeff Dean from Google:

In addition to bug fixes, several important new features and enhancements have made it into this release:

• Admin Client/Server API: intended for functionality which is required, but should be used sparingly (if at all), at the application level. This adds support for retrieval and update of metadata on remote nodes as well as streaming of keys and key/value pairs from one node to another.
• EC2 testing: a distributed system requires tests which involve multiple machines, contributed by Kirk True. Amazon’s EC2 web service allows us to provision and de-provision nodes programatically. The EC2 Testing Infrastructure allows for such tests to run on a regular basis along with other automated tests.
• Support for large lists and strings in the JSON serializer. Previously, the binary JSON serialization format limited us to maximum size of 32,768 for strings and lists (i.e. the maximum value of a signed 16-bit integer). The maximum size of a list or string is now 1,073,741,823 bytes.
• Experimental support for views. Views allow for computation to be moved close to the data. Suppose, for example, that we’re storing a serialized list as a value in a key/value pair and would like to append a single element. Normally, we’d have to transfer the entire list to the client, append the element, then transfer the modified list to the client. Views would mean a “put” operation now becomes a proxy for append a value to the list directly on the client. Note, this is feature is experimental, we can’t make any guarantees about the stability or performance of this feature; in addition, the API is also subject to change.
• Support for LZF compression, contributed by Ismael Juma and Tatu Saloranta.
• Interpolation search for read-only stores.
• A client for the Ruby programming language (using an experimental ruby protocol buffers gem), contributed by Claudio Cherubino.