In addition to bug fixes, several important new features and enhancements have made it into this release:

• Admin Client/Server API: intended for functionality which is required, but should be used sparingly (if at all), at the application level. This adds support for retrieval and update of metadata on remote nodes as well as streaming of keys and key/value pairs from one node to another.
• EC2 testing: a distributed system requires tests which involve multiple machines, contributed by Kirk True. Amazon’s EC2 web service allows us to provision and de-provision nodes programatically. The EC2 Testing Infrastructure allows for such tests to run on a regular basis along with other automated tests.
• Support for large lists and strings in the JSON serializer. Previously, the binary JSON serialization format limited us to maximum size of 32,768 for strings and lists (i.e. the maximum value of a signed 16-bit integer). The maximum size of a list or string is now 1,073,741,823 bytes.
• Experimental support for views. Views allow for computation to be moved close to the data. Suppose, for example, that we’re storing a serialized list as a value in a key/value pair and would like to append a single element. Normally, we’d have to transfer the entire list to the client, append the element, then transfer the modified list to the client. Views would mean a “put” operation now becomes a proxy for append a value to the list directly on the client. Note, this is feature is experimental, we can’t make any guarantees about the stability or performance of this feature; in addition, the API is also subject to change.
• Support for LZF compression, contributed by Ismael Juma and Tatu Saloranta.
• Interpolation search for read-only stores.
• A client for the Ruby programming language (using an experimental ruby protocol buffers gem), contributed by Claudio Cherubino.

• Extraction of entries/keys for backups
• Daily batch ETL (extraction, transformation, loading) to other systems for analysis (e.g. hadoop, search, etc…)
• Bulk loading of entries
• Migrating partitions
• Getting/Updating the cluster state/metadata

AdminClient APIs

These are the supported AdminClient APIs and their brief descriptions; please refer to the javadocs for a full reference. The AdminClient can be constructed given a voldemort server bootstrap URL or given a cluster object with information about nodes in the cluster.

• fetchEntries() : Provides a way to fetch all entries (key/value pairs) from a remote node belonging to any of the specified partitions.
  The call returns an iterator instantaneously which internally keep fetching new entries as new values are requested (aka streaming mode).
• fetchKeys() : Same as fetchEntries() but only returns the keys. The server side can be more intelligent and do optimization at storage level to for better performance than fetchEntries()
• updateEntries(): Provides a way to bulk load entries at a remote node, takes an iterator as parameter and start updating remote node with iterator values as they are streamed.
• migratePartitions() : Provides a way to migrate partitions from a remote node to another node by a third party. This API is used by the rebalancing system to start rebalancing operations at different nodes. The operation is started as an asynchronous operation at the updating remote node. A fetchEntries() request is started on the remote node, updating the values on that node. The status the of operation can be checked using the getAsyncRequestStatus() API.
• (Get/Set)RemoteMetadata(): These two APIs provide a way to get/set node metadata (cluster.xml, stores.xml, states) at individual remote nodes for cluster monitoring, status checks and upgrades.
• getAsyncRequestStatus(): Gets the status for async operation at the (remote) node. This allows the progress of an async operation (e.g. migratePartitions) to be monitored.
• waitForCompletion(): use exponential backoff to await the completion of an asynchronous request, simulating “blocking” behaviour.

Streaming support

Streaming is the efficient bulk transfer of entire partitions between machines. The intended uses are extraction of all keys (or all key/value pairs) for off-line manipulation (e.g. MapReduce processing) or backups, migration of partitions between machines and deletion of entries. Experimental support also exists for server-side filtering using a client-side supplied Java bytecode implementing the VoldemortFilter interface. The major application of the migration functionality is dynamic deletion and addition of nodes (”rebalancing”) which is presently in development.

For efficiency during streaming messages (containing key/value pairs) are sent one-after-anothere (over a buffered input stream), without waiting for acknowledgment (or forcing an early buffer flush); an “end-of-stream” message is sent signifying completion, after which is the buffer is flushed.

When the values are streamed from a store backed by the BerkeleyDB JE storage engine, the values are read from a cursor. At the present time, unfortunately, the cursors are only opened in key order. This presents an issue: when the entries aren’t present in BDB cache or the operating system page cache (this is guaranteed to occur when the dataset is larger than the machine’s physical memory), they have to be fetched from disk; if the cursor is opened in key order, unless the key order matches the disk order (i.e. ordered keys written sequentially) that means random seeks are made to access the values. Fortunately, BerkeleyDB, the operating system and the disk controller provide a way to schedule the seeks. In our experimental setup, on a machine with eight gigabytes of memory, a single 7200 RPM SATA disk and bdb.cache.size set to 3G were able to achieve the rate of ~1,200 entries a second: multiple times higher than would have been possible if random seeks were to be made for each entry. The distributed nature of Voldemort also means fetches of partitions can be performed from multiple nodes in parallel (e.g. fetch partition 1 from node a, partition 2 from b, etc…) distributing the random seeks across multiple nodes.

In addition, there is support for throttling the streaming operations. There are two settings stream.read.byte.per.sec and stream.write.byte.per.sec, responsible for throttling reads and writes respectively. The default settings for both are 10MB/second. This allows you to limit the amount of read and writes operations that are being done by the admin client API, as to prevent exhaustion of the seek capacity and starvation of the system of resources needed to handle routine requests.

Under the current release plan, patches (containing new features, enhancements to existing features and bug fixes) are accepted throughout on master branch; after thorough testing, these patches are merged. Automated testing runs whenever any commits are made, thus the master branch remains stable. Once a month, a “release candidate” is branched from the master.  No further patches are accepted into the release branch. After undergoing rigiorous testing (unit tests, as well as integration and performance tests passing), a release is made from this branch (i.e. versions are incremented). Release notes are kept, detailing changes brought on by this release (new features, issues/bugs fixed, new configuration options and especially any incompatibilities with previous releases). Any changes made on this release branch are merged back to the master.

As a young and agile open source project, we ask for and greatly appreciate community participation: we welcome all contribution, including patches and suggestions on how our release process could be tuned to better serve you.

Recently my NIO implementation of Project Voldemort’s socket server was merged into the master repository. I wanted to share the results of some testing I’ve done comparing the existing blocking I/O (BIO) implementation of the socket server vs. my NIO implementation. Hopefully this will encourage you to give it a try and let us know how it’s working for your workload.

Enabling the NIO Implementation

Enabling the NIO implementation is simply a matter of updating the configuration in Voldemort’s server.properties:

enable.nio.connector=true

This will enable the NIO socket server implementation.

Internally, the NIO implementation uses N threads, each with its own java.nio.channels.Selector object, used for readiness selection. The number of threads (N) is dictated by another configuration parameter:

nio.connector.selectors=2

Note: the nio.connector.selectors property is optional. If omitted, the number of threads used by the NIO socket server implementation defaults to the number of processors on the running system. Empirical testing found that a thread/Selector per processor worked most efficiently.

Test Environment

My testing environment is kept deliberately simple in order to encourage reproducibility. I’m running a single Voldemort instance on a server to which four client machines are pointing. I am simulating multiple clients by using multiple threads from each client.

In terms of the hardware involved, the server is an Intel Core 2 Duo 6700 (2.66 GHz) with 3.3 GB RAM. The client machines are Intel Core 2 Duo 4300s (1.80 GHz) with a full 4 GB of RAM each. The server and clients are separated by Gigabit Ethernet on a dedicated switch.

The server and client machines are running Fedora 10 64-bit. The JVMs are standardized to version 1.6.0_14. The code is version 0.52 from my git-forked “nio-server” branch. Besides the changes made to my branch, I have made two small uncommitted changes as detailed in the patch below[1]. Basically the uncommitted changes increase the heap size for both client and server to 3 GB and increase the client timeouts to 10 minutes(!).

Client Setup

To perform the tests I’m using the voldemort-remote-test.sh script on each of the four client machines, updated to use 3 GB heap as stated above. I updated the voldemort.performance.RemoteTest to set some timeouts very long (10 minutes). I simply averaged out the result from all four machines over 100 iterations of the test.

Server Setup

The server is set up to use either the BIO or NIO implementation depending on the value of “enable.nio.connector” in server.properties. The value of “storage.configs” was set to  “voldemort.store.noop.NoopStorageConfiguration,voldemort.store.bdb.BdbStorageConfiguration” as some tests used the no-op storage engine. When running the BIO implementation the value of “max.threads” was set as appropriate based on the total number of client threads. For the NIO implementation we run only two threads regardless of the fact that we’re serving tens of thousands of clients.

After each test the Voldemort server was shut down and restarted.

For both the client and the server I executed the command “ulimit -n 40000” prior to starting the JVM to run the tests in order to provide room for the socket file descriptors.

Test Results

So far I’ve run three tests comparing the BIO implementation to the NIO implementation. These all center around measuring the writes/second as reported by the voldemort-remote-test.sh script.

Transactions/second using No-op Storage Engine

The first test measured the transactions/second as the number of clients increased. The server is running the no-op storage engine to avoid skewing the results with BDB’s work (as is shown to be significant in the next test).

For this test, the invocation of voldemort-remote-test.sh provided these options:

Number of iterations: 100
Number of requests: 100,000
Operations: write and delete
Value size: 1024 bytes
Threads: as shown in the x-axis; each client ran ¼ of the threads

Here is a graph charting the results:

While running the BIO implementation simulating 6,000 clients, the client application started throwing lots of connection timeout errors upon starting up. After several minutes the clients would generally recover and complete the number of iterations. However, the reporting output from the client would show that the number of successful operations was often less than the expected. For example, the delete operation would report “99,984 things deleted” instead of 100,000. This connection timeout behavior explains the increase in speed shown by the BIO implementation. As a fraction of the clients hung (and eventually timed out), the remaining clients would process their work that much faster, producing a more favorable measurement. These connection timeout errors were not seen when running the NIO implementation until much later.

On the server, the load average (as measured by top) was about ½ the number of server threads, e.g. around 4,000 for 8,000 clients when running the BIO implementation. For NIO, it was always ~2. For 8,000 clients, the heap size went to 2 GB for BIO while NIO was about 700 MB. This is largely attributed to the per-thread stack size.

Individual iteration measurements for clients varied wildly with the BIO implementation. Unfortunately I didn’t measure this, but I did note several occasions where iterations would yield 15,000 writes/sec and the next iteration only 4,000 writes/sec. Deviation for the NIO implementation was observed to be minimal, however.

The data for the BIO implementation ends abruptly before recording the results for 10,000 connections since the clients hung until they were forcibly stopped (via kill -9). This was attributed to the inability of the BIO implementation to keep up due to the fact that it was pegged in garbage collection.

At 20,000 connections the NIO implementation did cause one client machine to timeout on connections, though it did recover. At 32,000 connections a noticeable drop in performance was noted and at 34,000 the clients hung until forcibly stopped. This too was attributed to garbage collection times.

Transactions/second using BDB Storage Engine

The next test also measured the transactions/second as the number of clients increased. This time, however, the server is running the BDB storage engine to see how the socket server implementation is affected.

For this test, the invocation of voldemort-remote-test.sh provided these options:

Number of iterations: 100
Number of requests: 100,000
Operations: write and delete
Value size: 1024 bytes
Threads: as shown in the x-axis; each client ran ¼ of the threads

Here is a graph charting the results:

As with the above test, around 6,000 clients with the BIO implementation the clients started throwing lots of connection timeout errors when starting up. At 8,000 clients this connection timeout behavior occurred for all four clients and explains the increase in speed shown by the BIO implementation. These connection timeout errors were not seen when running the NIO implementation until around 30,000 clients.

The BDB storage engine greatly affects the transactions/second that can be processed. My guess is that because the BDB code (and its threads) are competing for CPU time against thousands of other threads in the BIO implementation that it begins to become starved in terms of making progress. The NIO implementation, however, using a fixed two threads allows BDB to accomplish more work.

So even at a small number of clients, we achieve an increase of ~50-100% more transactions/second with the NIO implementation over the BIO implementation.

Transactions/second per Request Size

The next test also measured the transactions/second, but this time as the request size increased. The server is running the no-op storage engine to measure more clearly the network I/O.

For this test, the invocation of voldemort-remote-test.sh provided these options:

Number of iterations: 100
Number of requests: 100,000
Operations: write and delete
Value size: as shown in the x-axis
Threads: 100 total, each client ran 25 threads

Here is a graph charting the results:

It’s pretty easy to see how the request size has an obvious effect on the number of transactions. Please note that the request sizes in the x-axis are doubling rather than linearly increasing. As can be seen in the graph, this appears to be a matter of JVM throughput as opposed to the nuances of the socket server implementation.

Conclusion

The implementation of the socket server delivers on NIO’s promise of better scalability with regard to using asynchronous I/O and readiness selection over the blocking I/O and thread-per-socket approach.

A Big Surprise

For a long time my NIO implementation suffered from weaker performance than the BIO implementation. I had scoured the code looking for problems but found nothing obvious. I had used the examples that I’d seen posted everywhere wherein the server used a single Selector to process readiness selection and for each ready socket, handed processing of the thread off to a thread pool.

What I found during profiling of the server was that ~30% of the runtime was spent in updating the SelectionKey’s interestOps. Understandably when a SelectionKey updates its interestOps it requires acquiring a shared lock in the Selector implementation (at least for the 1.6 JDK on Linux). This caused a horrible serialization problem.

What I did instead was to move to a thread-per-Selector design wherein all of the I/O is done serially in a loop. There is literally one thread handling the I/O for tens of thousands of sockets. Because it’s done in the same thread, when the SelectionKey’s interestOps is updated, the lock is found to already be held, thus no contention.

Even while I’m writing this I’m dumbfounded how a single thread that includes I/O and processing the operation could outperform a multi-threaded design. I’ve looked and looked and tested and tested and it’s almost always faster. The code for the NIO implementation is there, take a look and let me know what I’ve overlooked.

1. Here’s an uncommitted patch that I used to increase the heap size and increase client timeouts:

diff --git a/bin/run-class.sh b/bin/run-class.sh
index ade56bb..a95d593 100755
--- a/bin/run-class.sh
+++ b/bin/run-class.sh
@@ -35,8 +35,8 @@ done
CLASSPATH=$CLASSPATH:$base_dir/dist/resources

if [ -z $VOLD_OPTS ]; then
-  VOLD_OPTS="-Xmx2G -server -Dcom.sun.management.jmxremote"
+  VOLD_OPTS="-Xmx3G -server -Dcom.sun.management.jmxremote"
fi

export CLASSPATH
-java $VOLD_OPTS -cp $CLASSPATH $@
\ No newline at end of file
+java $VOLD_OPTS -cp $CLASSPATH $@
diff --git a/bin/voldemort-server.sh b/bin/voldemort-server.sh
index 9878095..66a69e5 100755
--- a/bin/voldemort-server.sh
+++ b/bin/voldemort-server.sh
@@ -42,7 +42,7 @@ done
CLASSPATH=$CLASSPATH:$base_dir/dist/resources

if [ -z $VOLD_OPTS ]; then
-  VOLD_OPTS="-Xmx2G -server -Dcom.sun.management.jmxremote"
+  VOLD_OPTS="-Xmx3G -server -Dcom.sun.management.jmxremote"
fi

java $VOLD_OPTS -cp $CLASSPATH voldemort.server.VoldemortServer $@
diff --git a/test/integration/voldemort/performance/RemoteTest.java b/test/integration/voldemort/performance/RemoteTest.java
index 2a15955..29e945d 100644
--- a/test/integration/voldemort/performance/RemoteTest.java
+++ b/test/integration/voldemort/performance/RemoteTest.java
@@ -26,6 +26,7 @@ import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

import joptsimple.OptionParser;
@@ -132,6 +133,9 @@ public class RemoteTest {

System.out.println("Bootstraping cluster data.");
StoreClientFactory factory = new SocketStoreClientFactory(new ClientConfig().setMaxThreads(numThreads)
+                                                                                    .setConnectionTimeout(10, TimeUnit.MINUTES)
+                                                                                    .setRoutingTimeout(10, TimeUnit.MINUTES)
+                                                                                    .setSocketTimeout(10, TimeUnit.MINUTES)
.setMaxTotalConnections(numThreads)
.setMaxConnectionsPerNode(numThreads)
.setBootstrapUrls(url));

At Lookery, we have been working very hard to transform most of our data processing tasks into batch-oriented workflows in order to deal with growth. For example, we were already using Hadoop to compute our index and data files for our largest database, but the process of serving that information took place over too many network hops (load balancers, reverse proxies and Amazon S3). Therefore, as soon as I learned that Project Voldemort supported offline building of distributed stores, I decided to try it and we’re now running it in production. Please read the rest for an example walkthrough building a Voldemort read-only index and data files using Hadoop. The goal for this tutorial is to deploy a Voldemort cluster storing words and their counts calculated by the canonical Hadoop example: WordCount.

Step 1: Preferably build or download Voldemort:

    > git clone git://github.com/voldemort/voldemort.git
    > cd voldemort
    > ant

Step 2: Configure Cluster

Now let’s create a single node cluster for our wordcounts store. Create a directory and place the following files inside a directory named ‘config’ within that directory.

server.properties

    # The ID of *this* particular cluster node
    node.id=0

    max.threads=100

    http.enable=true
    socket.enable=true
    enable.readonly.engine=true

    file.fetcher.class=voldemort.store.readonly.fetcher.HdfsFetcher

cluster.xml

    <cluster>
      <name>wordcounts</name>
      <server>
        <id>0</id>
        <host>localhost</host>
        <http-port>8081</http-port>
        <socket-port>6666</socket-port>
        <partitions>0, 1</partitions>
      </server>
    </cluster>

stores.xml

    <stores>
      <store>
        <name>wordcounts</name>
        <persistence>read-only</persistence>
        <routing>client</routing>
        <replication-factor>1</replication-factor>
        <required-reads>1</required-reads>
        <required-writes>1</required-writes>
        <key-serializer>
          <type>json</type>
          <schema-info>"string"</schema-info>
        </key-serializer>
        <value-serializer>
          <type>json</type>
          <schema-info>"int32"</schema-info>
        </value-serializer>
      </store>
    </stores>

Let’s now make sure that everything is working as expected. Voldemort will automatically create a blank read-only store in a subdirectory called data if an existing is not found.

    > $VOLDEMORT_HOME/bin/voldemort-server.sh . &> /tmp/voldemort.log &
    > $VOLDEMORT_HOME/bin/voldemort-shell.sh wordcounts tcp://localhost:6666
    Established connection to wordcounts via tcp://localhost:6666
    > get "voldemort"
    null

Step 3: Prepare your input data to organize into a word and count sequence.

Before you continue, you must already have Hadoop configured in either pseudo-distributed or fully-distributed mode. Unfortunately, I’m going to punt on this part of the setup because there’s plenty of good documentation on the Hadoop wiki itself. Once you have your cluster running, let’s proceed to run the wordcount example.

    > bin/hadoop dfs -copyFromLocal <local-dir> <hdfs-dir>
    > bin/hadoop jar hadoop-*-examples.jar wordcount [-m <#maps>] [-r <#reducers>] <in-dir> <out-dir>

We now have a tab-separated records that contain the word and its count: “<word>TAB<count>” stored in the <out-dir> directory.

Step 4: Use your favorite Java IDE/Editor to implement your Hadoop mapper

Project Voldemort provides you with most of the code necessary to build the store. The only missing part is how to extract the key and value objects from your input files. Since our WordCount example outputs our data in a very simple key,value format, our mapper only needs to split the actual line on the TAB character and return a String object for the key and an Integer object for the value just as declared in our store definition inside stores.xml. You will need to update your CLASSPATH to point to the voldemort jars and their dependencies in order for this to compile. Upon completion, you should export this class into a jar called wordcount-mapper.jar and place it inside a ‘lib’ directory right along your ‘config’ directory.

    package com.lookery;

    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;

    import voldemort.store.readonly.mr.AbstractHadoopStoreBuilderMapper;

    public class HadoopStoreMapper extends AbstractHadoopStoreBuilderMapper<LongWritable, Text> {

        @Override
        public Object makeKey(LongWritable key, Text value) {
            return value.toString().split("\t")[0];
        }

        @Override
        public Object makeValue(LongWritable key, Text value) {
            return Integer.parseInt(value.toString().split("\t")[1]);
        }
    }

Step 5: Build your read-only store from the command-line

The next step is for you to call Project Voldemort’s shell script as shown in the example below. Most of the parameters should be self-explanatory, if not you can always call the script with ‘–help’ to get full details for all available parameters. However, it might be worth explaining a couple in this example. First, let’s look at chunk size. The main reason for using Hadoop to build large read-only stores is because of its ability to run tasks in parallel. But to help you take advantage of Hadoop, Project Voldemort builds the index and data files for each node in the cluster in chunks in order to let the developer maximize the number of reducers his Hadoop cluster dedicates to building the distributed store. You should play with the chunksize until it gives you a number within a small multiple of your maximum reducer availability. In this case, I set it to 1 gigabyte. The second option worth mentioning is the replication factor. This setting will let you create redundancy in your storage so your cluster can continue serving values even after some node failures. Please see the rest of the documentation to choose what is the best replication factor in your application scenario.

    > $VOLDEMORT_HOME/bin/hadoop-build-readonly-store.sh --input <input_dir> \
        --output wordcounts --tmpdir tmp-build --mapper com.lookery.HadoopStoreMapper \
        --jar lib/wordcount-mapper.jar --cluster config/cluster.xml \
        --storename wordcounts --storedefinitions config/stores.xml \
        --chunksize 1073741824 --replication 2
    09/06/17 20:24:38 INFO mr.HadoopStoreBuilder: Data size = 17934, ...
    09/06/17 20:24:38 INFO mr.HadoopStoreBuilder: Number of reduces: 1
    09/06/17 20:24:38 INFO mr.HadoopStoreBuilder: Building store...
    09/06/17 20:24:38 INFO mapred.FileInputFormat: Total input paths to process : 1
    09/06/17 20:24:38 INFO mapred.FileInputFormat: Total input paths to process : 1
    09/06/17 20:24:38 INFO mapred.JobClient: Running job: job_200906171849_0002
    09/06/17 20:24:39 INFO mapred.JobClient:  map 0% reduce 0%
    09/06/17 20:24:44 INFO mapred.JobClient:  map 100% reduce 0%
    ...

Step 6: Ask the cluster to fetch the newly built read-only in Hadoop

Now the easiest part is to ask each node in the cluster to fetch its index and data chunks and only after they have succeeded, then they will all atomically swap their stores to the latest version. This has to be one of the coolest features in the read-only store. We hope you like it too. There are a couple of parameters available to the server.properties file that could help you further enhance this capability. The first is that you can specify a temporary folder for downloading the files from Hadoop HDFS (hdfs.fetcher.tmp.dir). This is important to make sure that both the read-only data store and the temporary folder are in the same device in order to avoid an extra copy of the downloaded files and make the swap truly atomic as permitted by the underlying filesystem. The second option (fetcher.max.bytes.per.sec) is to signal the HdfsFetcher to throttle its download rates in order to avoid interference with the online requests from Voldemort clients. There are also some other options for the swap-store.sh command such as timeout that could help you deal with large download periods from the cluster nodes. But other than that, it should be straight forward to update your online read-only store using this command.

    > $VOLDEMORT_HOME/bin/swap-store.sh --cluster config/cluster.xml \
        --file hdfs://localhost:54310/user/${user.name}/wordcounts --name wordcounts
    [2009-06-17 23:35:23,946] INFO Invoking fetch for node 0 for hdfs://localhost:54310/user/elias/node-0
    [2009-06-17 23:35:24,045] INFO Fetch succeeded on node 0 (voldemort.store.readonly.StoreSwapper)
    [2009-06-17 23:35:24,046] INFO Attempting swap for node 0 dir = /tmp/hdfs-fetcher/hdfs-fetcher/node-0
    [2009-06-17 23:35:24,060] INFO Swap succeeded for node 0 (voldemort.store.readonly.StoreSwapper)
    [2009-06-17 23:35:24,060] INFO Swap succeeded on all nodes in 0 seconds.

Step 7: Verify that your data was loaded successful

    > bin/voldemort-shell.sh wordcounts tcp://localhost:6666
    Established connection to wordcounts via tcp://localhost:6666
    > get "voldemort"
    version(): 2

We have been running at Lookery with this setup for almost a month now and have been very pleased the results. It allowed us to have a Voldemort cluster that is always refreshing with the new data without manual intervention. If you have any corrections or suggestions on how we can improve both the batch indexer or this tutorial, please don’t hesitate to email the mailing list.

The difficulty in these systems comes with the fact that large amounts of data need to moved around every day. Thus although hundreds of gigabytes or terrabytes of data are not to difficult when sitting still in a storage system, the problem because much, much harder when it must be transformed to support quick lookups and moved between systems on a daily basis.

This post describes the system we built to deploy data to the live site using our key-value storage system, Project Voldemort.

Why do we end up with so much data? The size of the output is usually determined by the quantity of something on the site: we might compute something for each member profile, each question that gets asked, each news article that is posted, etc. These jobs may process a lot of data, especially if they involve any of the very large logging data streams, but the results, though large, are manageable. We have a second kind of job that is at least as common and produces results for each pair of users, or each pair of companies, or, say, the relationships between users and questions, or between the many other types of content on our site. As you might imagine the number of interesting pairs of items is much larger than the number of actual items (it isn’t as large as the square of the number of items, since most pairs aren’t interesting, but it is still huge). This seems to be a natural use case for social networks where the relationships are of central importance. Previously we did not need to confront this problem both because our data size was smaller, and also because our ability to produce large offline datasets was limited by computation constraints. Hadoop has been quite helpful in removing scalability problems in the offline portion of the system; but in doing so it creates a huge bottleneck in our ability to actually deliver data to the site. As is often the case, removing a bottleneck in one area creates a new bottleneck somewhere else.

To solve this problem we spent some time thinking about how to build support for large daily data cycles. Voldemort was designed to support fast, scalable read/write loads, and is already used in a number of systems at LinkedIn. It was not designed specifically with batch computation in mind, but it supports a pluggable architecture which allows the support of multiple storage engines in the same framework. This allows us to integrate our fast, failure-resistent online storage system, with the heavy offline data crunching running on Hadoop.

Here is a picture of what our world looks like:

Some existing approaches

There are plenty of other ways to approach this problem, but no one we talked to had a good solution. We saw many variety of things being done, including pushing static text files by hand, FTPing giant XML files or doing JDBC batch inserts in an (Oracle) DB. None of these are really good approaches to the problem, since they typically have one of two common problems. The first is that the data transfer is centralized, creating an un-scalable bottleneck in the delivery of the data. The second is that the process of building the lookup index (generally a btree) is happening on the same live server that is serving lookups. This is a big problem since building a large index is a huge and computationally intense operation that may take hours, and by doing this on the live server we are effectively mixing this huge throughput-oriented operation with short-latency sensitive lookups, generally with poor results for your users.

So what alternatives are there?

The best online system for data lookups right now is memcached. Memcached is stable and has excellent performance for common caching needs. The obvious problem with memcached are the “mem” and the “cache” parts. Memcached is all in memory so you need to squeeze all your data into memory to be able to serve it (which can be an expensive proposition if the generated data set is large). In addition memcached is a cache, so if you need to restart your servers then your data will disappear and need to be re-pushed! Another problem is the apparent lack of batch set operations. Without this the majority of time will inevitably be spent on unnecessary network round-trips no matter how efficiently we implement them. We could easily build a map reduce job to do this in parallel, but that only works around the underlying weakness in the per-node transfer rate.

The next best online system is MySQL. MySQL can avoid the one-round-trip per insert problem by doing batch inserts, but even that seems to give rather low throughput. MySQL’s InnoDB table format has too high space overhead to make it a real competitor. However MySQL has a very slim and simple MyISAM format. MyISAM isn’t used as much for normal read/write usage since it uses a global table write lockand lacks many transactional features, but this isn’t a problem for read-only usage since it is write-free. MySQL also supports an optimized “load data infile local” statement that provides bulk load capability. This is an extremely important feature for a disk-based storage format in this use case–building a 100GB index can not be done effectively as a sequence of b-tree updates that incrementally re-arrange data as the they go because the total IO casued by all the little updates is extremely high. To avoid this the tree needs to do a batch build that builds as much of the tree at once as possible, and this is exactly what the “load data” statement does. All-in-all MySQL is slim, quick, and generally the best off-the-shelf solution to this problem we have seen. Still to make this build effective you need a ton of memory, and it will lock the table for the duration of the build. This means that if you are running this on your live servers they will be extremely heavily worked for the duration of the load (which can easily take hours). Not to mention that MySQL provides little in the way of ability to parallelize this, making constructing a system on top of this a difficult proposition.

Clearly building an index like this is an offline operation and should not be done on a server that is serving live traffic as it will likely choke the CPU and IO resources from serving the live requests. In principle this is possible as MySQL (rather frighteningly) seems to allow you to just copy the files for a database into the database directory of a running server which will immediately make the table appear available without restarting. But this would mean maintaining a whole separate cluster of MySQL servers just for the purpose of index building as well as devising some way of parallelising this process. Finally a practical point is that you will likely have to write the data to disk multiple times–once to copy it to the server as a text file, then again as it is built as a database, and finally a third time if it copied to a live server (not to mention that fact that MySQL unfortunately seems to make its own internal copy of the data as well when you build the index to support its transactional requirements). Since the load data statement doesn’t seem to support compression, storing your data as CSV is a rather large blow-up. These things don’t seem like they should be a big problem, but when your 400GB data set turns into a 1,200GB dataset because all the numbers are in ASCII, and this file is then copied multiple times, that creates a serious problem.

Requirements for a better solution

These alternatives weren’t attractive, so we thought through what would be needed to do a good job with this problem. We came up with the following things:

1. Protect the live servers. Uploading a new data set can’t impact the services relying on the data. We want the upload of new results to go as fast as possible but no faster. This means moving as much computation out of the online system as possible, and guaranteeing the live servers are not negetively impacted.
2. Horizontal scalability at each step. Hadoop gives us a scalable approach to the build, and Voldemort gives a scalable system for the lookups. The trick is just ensuring that there is no centralized bottleneck in the process.
3. Ability to rollback. Like any code, the processes that generates the data may have some kind of error or bug that leads to generating corrupt data, but unlike most code problems, fixing things may not be so quick. Since the processes may take many hours to run, and the data automatically goes live without human perusal, this kind of failure can leave us in a bad position. It may take hours to rerun (or for some very computationally intense processes, days), and we will could be stuck with the bad data until we manage to fix the bug, rerun everything, and re-push the fixed data. This is clearly unacceptable. As a result we would like to retain multiple copies of the data set, one for each of the last N pushes (where in the common case N = 1) so that we could revert to this known good state. This allows us to have a constant time rollback to a previous data set.
4. Failure tolerance. This is where the Voldemort consistent hashing comes in to play–a server failure in the live system will redirect 1/K of that server’s traffic to each of the remaining servers without impacting the client. We also get similar failure tolerance in the build from using Hadoop.
5. Support large ratios of data to RAM. The original problem we are trying to solve is that the data size is very large so we need to design accordingly. Improving performance in the case where the data is all in memory is not terribly valuable, the focus is on supporting a data size significantly larger than memory on each node.

Our approach

Bhupesh, Elias, and I toyed with solutions to these requirements, and here is the design we came up with.

One thing was clear, the Hadoop cluster is the natural place for the index build to occur. Hadoop is where the data is when the processing is done, and the goal of these machines is to run at full utilization so however computationally intense the build process is, it will not be a problem.

For the live system we wanted to adapt our key-value system, Voldemort. To do this we wanted to add an on-disk structure optimized for access to very large read only data sets we could deploy in batch. In particular we wanted some kind of simple file-based format we could stream to the servers to avoid doing many network round trips during the deployment. Ideally we should be able to deploy data at the rate possible by the network or disk system of the Voldemort and Hadoop clusters.

In early versions of the storage engine we toyed with different lookup and caching structures. But some simple benchmarking revealed this to be a rather academic exercise. The fundamental fact of filesystem access is that you may or may not be accessing the underlying disk depending on whether your request can be served by the OS’s pagecache or not.  A pagecache hit on an mmap’d file takes less than 250 nanoseconds but a page miss is around 5 milliseconds (a mere twenty thousand times slower). Any fancy data structure we build is likely to reside in-memory. Hence it would only help the lookups for things that would be in page cache anyway (since the process of loading them into memory would put them there) and so lookups on these would be fast no matter what. And worse this in-process lookup structure will likely steal memory from the pagecache to store its data, and since this will duplicate things in the pagecache it is extremely inefficient. Thus even if we manage to improve the lookup time for the things in our process memory, it is already quite low; and by doing so we use up memory that moves more requests out of the ns column and into the ms column. In short, Amdahl wins again.

To take advantage of this we have a very simple storage strategy that exploits the fact that our data doesn’t change–all we do is just mmap the entire data set into the process address space and access it there. This provides the lowest overhead caching possible, and makes use of the very efficient lookup structures in the operating system. Since our data is immutable, we don’t need to leave any space for growth and can tightly pack the data and index. Since the OS maintains the memory it can be very aggressive about this cache, and indeed it will attempt to fill all free RAM at all times with recently used pages. In comparison Java is a very inefficient user of memory since it must leave lots of extra space for garbage collection, etc. Plus anyone who has gotten intimate with Java GC tuning will not object to moving things out of the Java heap space.

How data is stored

Sometimes it’s nice to know what is going on under the covers. The data for a store named my_store would consist of the following files:

my_store/
  version-0/
    0.index
    0.data
    ...
    n.index
    n.data
  version-1/
    0.index
    0.data
    ...

As you can see a store is just a directory of simple files. The .data files contain variable length values and the .index files contain the lookup structure necessary to map keys to values. In principle only one .index and .data file would be needed, but since writing a file is inherently single-threaded we break it into chunks numbered 0 through n to allow greater parallelism in the build. These chunks are then grouped into version directories containing a complete version of the data, with version-0 containing the current live data set.

Deploying a new version of the data consists of adding a new directory and renaming the existing ones. Storing multiple copies of the data is clearly a huge waste of space, but this is not too important as inactive files use no pagecache space just disk space. Small low latency reads on a huge data set will be largely seek bound, so we are going to need a lot of disk spindles no matter how we store things. Hard drive space is fairly cheap, so getting slightly larger disks to store additional copies is not a big problem.

To reduce the size of the file pointers and to work around limitations in Java’s mmap implementation, we limit chunk files to a maximum size of 2GB, so a reasonably sized store will consist of tens or hundreds of chunks per Voldemort node.

The order in which the values in the .data files are are stored is not important. Each value is prefixed by a 4 byte length indicating how many bytes to read. Each value is uniquely identified by the offset in the file at which its 4 byte length begins. The index contains 16 byte MD5 hashes of the keys along with the associated 4 byte position offset of the value in the data file. Because we hash keys, each key/value pair we store has a fixed overhead of exactly 24 bytes in addition to the length of the value itself. Furthermore this can be stored very efficiently as we can calculate the location of the ith index value as 20 * i. All lookups are positional; no internal pointers are needed within the index to locate entries.

The question is how to structure the keys in the index for quick lookups? A page- or block-organized tree is a good data structures if the data does not fit in memory. But this complicates both the lookups (which would need to be block aware), and the build process. In particular we want to perform our build in hadoop, which means that we will be limited to the amount of memory available to the mapper and reducer tasks which may leave only a few hundred megabytes for the build–if this does not fit our index data then we will have to perform some kind of external tree build. However on consideration we realized that since the index contains only 20 bytes per key even a very moderate amount of memory can hold several hundred million entries. Given this low overhead, very likely the whole index can (and should) be in memory (pagecache, not java heap), and so organizing the data by block or page is not really very important. As a result we greatly simplified our design–we just store the index entries in sorted order by md5 hash of the key.

A lookup in the store proceeds as follows:

1. Calculate the MD5 of the key
2. The first 4 bytes of this md5, modulo the number of chunks, is the chunk number to search in
3. Do a binary search for the key md5 in that chunk’s .index file to get the position of the value in the data file
4. Finally read the appropriate number of bytes for the value from the data file starting at the given position

The code for this storage engine is quite simple, only a few hundred lines, with the distribution and fault tolerance–the hard problems–being provided by the rest of Voldemort.

Binary search is not a very efficient algorithm for finding the location of the data. Most of the time this is not important since the index is in memory and so data access time dominates, but there are two cases that could be improved. The first is the case where all data and index fit entirely in memory. With very small keys, a chunk might have an index with, say, 100 million entries, which means a binary search does 27 key reads and comparisons and a single data read. In this case the cost of the search will dominate. Another suboptimal case is when we have an entirely uncached index. We explicitly transfer index files last in the data deployment to avoid this case, however in the case of rolling back to a previous index version it is unavoidable. To page the 100 million entry index for a chunk into memory will require 500k page faults no matter what the structure is. However it would be desirable to minimize the maximum number of page faults incurred on a given request to minimize the variance of the request time. In this case a page-organized tree, where each parent had 204 20 byte children, could do only log_204(100 million) = 4.5 page faults in the worst case and would be superior.

To resolve these cases we are working on an improved search algorithm which takes into account the uniformity of the key distribution, whch results from the fact that MD5 is (somewhat) cryptographically secure and so its keys are uniformly distributed. Rather than always beginning with a comparison to the middle entry such an algorithm would use the uniformity of the key distribution to compute the expected quantile of the key being looked up attempting to jump immediately to the correct location. If we can get a reliable implementation this promises to greatly improve the number of both page faults and comparisons needed in these corner cases.

Index Building

To build these store files we created two programs: a single-process command-line java program and also a distributed Hadoop-based store builder. The single process program uses a simple external sort to build the index files. Since this is a centralized process it is only useful for small data sets, testing, or one-time builds.

The Hadoop-based store builder is actually substantially simpler than the single-process builder as it leans heavily on Hadoop’s native capabilities to do its work. The store building processes proceeds as follows. An user-extensible Mapper extracts keys from the source data. This mapper can be parametrized to work with different InputFormats, and provides hooks to allow custom ways to construct the key and value from the data. A custom Hadoop Partitioner then applies the Voldemort consistent hashing function to the keys, and assigns all keys mapped to a given node and chunk to a single reduce task. The shuffle phase of the map/reduce copies all values with the same destination node and chunk to the same reduce task. Thus each of the reduce tasks will create one .index and .data file for a given chunk on a particular node; and as a result the number of chunks specified in the configuration acts as a parameter to control the parallelism of the build. These values are then sorted by Hadoop in order to group them by key for reduce. Each reduce task copies the key/value pairs it is given into a pair of .index and .data files in sorted order to build its store chunk.

Data deployment

It is important that we be able to swap in a complete data set all at once without any downtime or impact to the live cluster. As described above, multiple data versions are kept in the version- subdirectories, but only version-0 is used for serving data. Versions 1 through n are effectively backups. When a new data version is deployed, the version number of each existing data set is incremented, and the new set becomes the new version-0. To perform this swap a simple reader/writer lock is used to halt readers, each directory version-i is moved to version-(i+1), and the new data is moved to version-0 and the store is reopened and unlocked using this new dataset. Since only file renames are used, this is an O(1) operation, and in practice the whole procedure seems to complete in a few milliseconds irrespective of file size. The deleting of the N+1st version is prolonged until after the lock is released as delete may not be an O(1) operation, and may take over a minute on a filesystem that lack extents such as ext3.

The actual method for transferring data is pluggable. The original prototype used rsync in hope of efficiently supporting the transferring of diffs. However, this has two practical problems. The first was that the rsync diff calculation appears to be quite expensive, and half of the expensive calculation is done on the live server. Clearly if we want to do diffs, that too should be done on the batch system (Hadoop) not the live system (Voldemort). In fact due to this heavy calculation rsync was actually slower than just copying the whole file, even when the diff was rather small (though presumably much more network efficient). The more fundamental problem was that using rsync required copying the data out of HDFS to some local unix filesystem–which had better have enough space!–to be able to run rsync. This copying took as long as the data transfer to Voldemort, and meant we were copying the data twice.

To avoid these problems we switched from a push model to a pull model. It was important that we could schedule the transfer from the batch system to run automatically when the build completed successfully, so this took the form of a RESTful fetch command which triggers the Voldemort servers to fetch the data directly from HDFS. This mechanism is pluggable and a third party can provide an alternate implementation of the fetch command to support non-HDFS based mechanisms.

HDFS provides great throughput and seems to be able to max out the write capabilities of the Voldemort node. This is a blessing and a curse. Anyone who has lived with JDBC-based data transfer and seen it bottleneck on a measly few hundred KBs/sec will be overwhelmed with joy. But once again, removing performance problems in one area creates performance problems elsewhere: the high rate of data transfer to the live servers, even without any index building, can potentially starve live requests. However, in this model, where the server controls the pull, the Voldemort nodes can be configured to throttle itself to a fixed MB/sec limit so as not to overwhelm the I/O capabilities of the local Voldemort node. We have implemented a Voldemort configuration property, fetcher.max.bytes.per.sec, that controls this rate.

We have provided a driver program which initiates this fetch and swap procedure in parallel across a whole Voldemort cluster. In our tests this process can reach the I/O limit of either the Hadoop cluster or the Voldemort cluster.

Some benchmarks

There are two things to benchmark: the build time for a store in Hadoop and the request rate a node can sustain once live. We completed our benchmarks on EC2, since this is an easy way to get big clusters up and running for a quick test. Hopefully this will aid in making the results reproducible by others interested in testing different scenarios. We used extra large instances for both Hadoop and Voldemort as these most closely match our own hardware, and we used the Cloudera Hadoop AMI to get the test cluster up and running quickly.

Benchmarking anything that involves disk access is notoriously difficult because of sensitivity to three factors:

1. The ratio of data to memory
2. The performance of the disk subsystem, and
3. The entropy of the request stream

The ratio of data to memory and the entropy of the request stream determine how many cache misses will be sustained, so these are critical. A random request stream is more or less un-cachable, but fortunately almost no real request streams are random. They tend to have strong temporal locality which is what page cache eviction algorithms exploit. So for our testing we can assume a large ratio of memory to disk, and test against a simulated request stream to get performance information.

The performance is still very sensitive to the quality of the disk subsystem used for the Voldemort nodes. A live system like this will do lots of quick seeks with relatively small reads and will likely be bound by the seek time of the hard drive and the number of drives. The drives on the EC2 machines are fairly weak and not configured with RAID so they are not optimal if you are purchasing hardware, and in our tests all the processes we benchmark are IO bound. To help make sense of all these variables we provide a comparison to MySQL’s performance on the same tasks on the same hardware.

We are not aware of an existing system that does full build and data deployment in parallel, so there no direct comparison possible. But any build process will consist of three stages: (1) partitioning the data into seperate sets for each destination nodes, (2) gathering all data for a given node, and (3) building the lookup structure for that node. We can only compare results for the actual build (i.e. part 3) with MySQL as there is no off-the-shelf method for (1) and (2).

For our tests the keys are integers in ascii form. The values are meaningless 1024 byte strings.

Build Time

We tested the Hadoop build for a variety of store sizes. This time is the complete build time including mapping the data out to the appropriate node-chunk, shuffling the data to the nodes that will do the build, and finally creating the store files. In general, the time was roughly evenly split between map, shuffle and reduce phases. The number of map and reduce tasks are a very important parameter, as experiments on a smaller data set showed that varying the number of tasks could change the build time by more than 25%, but due to time constraints no attempt was made to optimize these, we just used whatever defaults Hadoop produced. Here are the times taken:

• 100GB: 28mins (400 mappers, 90 reducers)
• 512GB: 2hrs, 16mins (2313 mappers, 350 reducers)
• 1TB: 5hrs, 39mins (4608 mappers, 700 reducers)

To compare the build time we created a RAID 10 array on a single extra large instance, and did a build using one node’s worth of data (100m keys). This  process to 6 hours and 3 minutes to build the 100GB table for single node. Assuming similar performance for partitioning and copying data around this would indicate a complete build time of almost 8 hours per destination node. But this comparison ignores the time necessary to extract the data from the source system and convert it to CSV format for loading. And, of course, this neglects the additional benefits of Hadoop for handling failures, dealing with slower nodes, etc.

In addition, this process is scalable: it can be run on a number of machines equal to the number of chunks (700 in our 1TB case) not the number of destination nodes (only 10).

Data transfer between the clusters happens at a steady rate bound by the disk or network. For our Amazon instances this is around 40MB/second.

Online Performance

Lookup time for a single Voldemort node compares well to a single MySQL instance as well. To test this we ran local tests against the 100GB per-node data from the 1 TB test. This test as well was run on an Amazon Extra Large instance with 15GB of RAM and the 4 ephemeral disks in a RAID 10 configuration. To run the tests we simulated we simulated 1 million requests from a real request stream recorded on our production system against each of storage systems. We see the following performance for 1 million requests against a single node:

These numbers are both for local requests with no network involved as the only intention is to benchmark the storage layer of these systems.

How to actually use it

The code is all checked in to the main project repository on github. The commands for building a store, and executing a swap can be found under the bin/ directory. Elias has written a blog entry on how to use these, and how he has put this system into action at Lookery.

Future work

Nothing is ever finished, and below are a few of the ideas we didn’t quite get to. There are a lot of huge performance wins that exploit the immutable nature of the data that we have not yet taken advantage of. If any one is interested in playing with one of these problems here are a few ideas. LinkedIn is also looking for engineers to work on Project Voldemort full time, so if that sounds interesting send us a resume.

Incremental data updates

Despite the problems with rsync, incremental data pushes would be quite a big improvement for the case where the data changes by only 5%. This is a common case for a job that runs daily to recompute a large set of values. Getting efficient incremental performance is a harder problem than it sounds. We have never gotten a production system that will do this well in our past attempts: rsync didn’t seem to work well, MySQL’s load data performance is destroyed by pre-existing unique indexes, and Oracle insert/update is slower than a complete transfer and rebuild for anything but the most minor of changes.

There are two ways we can think of to support this. The first is the easiest to implement, and just consists of creating a diff file in much the same way that Unix diff or rsync , and using it in combination with the existing data on the live server to create the new set (rather than deploying everything each time). Since computing the diff turns out to be a very computationally intensive task for a large file, this work must be done in the offline Hadoop system. There is little point in trying to do incremental updates to the .index files as these are comparably small, and the changes are liable to be randomly interspersed throughout–so the diff won’t be much smaller than the original file. The .data files, however, do not have any inherent order so all the new data could be placed at the end of the file allowing for extremely efficient diffs. The “patch” could be applied in the process of doing the fetch, so that existing data segments would be read from the current version-0 dataset and new segments would be read from the HDFS diff file.

The above strategy reduces the network transfer necessary and could be run in steady state each day. However it does still require writing the complete data set to disk for each deployment, rather than writing the smaller diff only. One strategy that could avoid this would be to create a seperate set of .index and .data files for each day and store each days data seperately on the live system. A naive lookup would have to check each of these directories from latest to oldest to retrieve a value. However a more sophisticated approach could keep a Bloom filter tracking which keys are in each day’s patch. This would give a quick way to determine which files need not be searched without actually performing the search (with high probability).

Improved key hashing

The consistent hashing algorithm Voldemort uses has the nice property that N copies of a key are more or less randomly distributed over the cluster. As a result a failed node redistributes load evenly to the remaining nodes. This is an essential property to be able to tolerate node failure. However this algorithm has the unfortunate property that the data on each machine is different, and as a result the data built for each machine is unique. This means that a replication factor of two, doubles the size of the data that must be built. A hashing algorithm tuned to this use case could avoid this problem by replicating at the chunk level. This would provide less fine-grained load distribution since each chunk of data would be fully replicated on N machines, but would avoid the blow-up due to replication factor. Voldemort supports pluggable hashing algorithms so this should not be too difficult to implement, but didn’t make the cut for our first attempt.

Compression

Because the data is known upfront and does not change, it is a good target for compression. LZO compression or another compression algorithm tuned towards fast decompression speed could improve performance by reducing IO.

Better indexing

Fancier index structures are another area for improvement. We think the probabilistic binary search will prove to be a very effective approach, but since we haven’t implemented it yet it is worth considering a few other approaches.

The idea of a 204-way page-aligned tree instead of a binary tree was mentioned above. Each set of 204 20-byte index entries would take 4080 bytes which with 16 bytes of padding would then be exactly page aligned for a 4k page. This would mean the first 4k page of the index file would contain the hottest entries, the next 204 entries would contain the next hottest, and so forth. Thus even though the number of comparison necessary to locate an entry would not asymptotically improve the maximum number of page faults necessary to do an index lookup would decrease substantially in practical terms (to 4 or 5 for a large index).

This is not the optimal tree structure for an immutable tree such as ours, though. A much better approach to this problem was brought up by Elias who was familiar with the literature on cache-oblivious algorithms, and was aware of a cache-oblivious structure called a van Emde Boas tree. Cache oblivious algorithms uses a data structure that recurses in on itself in a way that requires no assumptions or special treatment for page sizes or CPU caches to get optimal cache performance (under some reasonable assumptions). There is a well developed set of cache-oblivious data structures for dealing with disk-based tree lookups. These algorithms manage to nicely utilize CPU cache as well, all without explicit assumptions about the memory hierarchy.

Still another alternative would be an on-disk hash-based lookup structure. Such a structure can reduce the number of required comparisons in a lookup, though as with a tree, its creation could be difficult. The trade-off between extra space used in the hashing and the collision rate is well known. At the far end of this spectrum is the minimal, perfect hash which is a function that hashes a fixed set of N keys to exactly N hash table slots. This structure would seem to be optimal for the lookups since it guarantees that we will require no more than one lookup to find the location of the data (indeed we could entirely avoid storing the hash value itself reducing the index entry size to only 4 bytes for the position). The hash function itself requires only about 3 bits per entry to be stored once it has been found. Computing these hashes can be difficult, though, so only a real implementation would show if the superior lookup time was justified by the possible increase in build time. There is an off-the-shelf MPH implementation from the author of mg4j but we have not yet investigated the feasibility of this in much detail.

You can check out the site code here: http://code.google.com/p/project-voldemort/source/browse/#svn/site/www.

Patches gladly accepted.