greg/hanoidb

Fork 0

HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.

Find a file

Kresten Krab Thorup 8f1600b41a Fold back pressure, step 2 This makes fold-from-snapshot use the back pressure model of doing plain_rpc:call to the merge worker delivering chunks of 100 KVs. The back pressure is entirely internal to hanoi, designed to ensure that the process that merges fold results from the individual levels is not swamped with fold data. Folds with a limit < 10 still do "blocking fold" which is more efficient and uses fewer FDs, but blocks concurrent put/get operations.		2012-05-05 18:53:02 +02:00
doc	sample 20 min run (MacBook Air/SSD)	2012-04-26 01:51:08 +02:00
include	Introduce plain_rpc	2012-04-26 00:49:57 +02:00
src	Fold back pressure, step 2	2012-05-05 18:53:02 +02:00
test	Enable quick check for backend tests	2012-05-02 17:11:08 +02:00
.gitignore	Provide a Makefile for running dialyzer.	2012-01-06 21:22:23 +01:00
DESIGN.md	Cleanup	2012-04-24 10:37:45 -04:00
enable-hanoi	Rename "lsm-btree" to "hanoi".	2012-04-21 15:20:39 -04:00
LICENSE	* Changed "lookup" to "get" just because	2012-04-15 10:35:39 -04:00
Makefile	Rename "lsm-btree" to "hanoi".	2012-04-21 15:20:39 -04:00
README.md	Config option {merge_strategy, fast\|predictable}	2012-05-01 16:27:06 +02:00
rebar	Initial work-in-progress	2012-01-04 15:05:31 +01:00
rebar.config	Add riak_kv as dependency	2012-05-02 17:13:58 +02:00
TODO	Update README/TODO	2012-04-28 18:42:04 +02:00
visualize-hanoi.sh	Refactor merge work computation	2012-04-30 23:34:27 +02:00

README.md

Hanoi Ordered Key/Value Storage

Hanoi implements an ordered key/value storage engine, implemented using "doubling sizes" persistent ordered sets of key/value pairs, much like LevelDB.

Here's the bullet list:

Insert, Delete and Read all have worst case log₂(N) latency.
Incremental space reclaimation: The cost of evicting stale key/values is amortized into insertion
- you don't need a separate eviction thread to keep memory use low
- you don't need to schedule merges to happen at off-peak hours
Operations-friendly "append-only" storage
- allows you to backup live system
- crash-recovery is very fast and the logic is straight forward
- All data subject to CRC32 checksums
Supports efficient range queries
- Riak secondary indexing
- Fast key and bucket listing
Uses bloom filters to avoid unnecessary lookups on disk
Efficient resource utilization
- Doesn't store all keys in memory
- Uses a modest number of file descriptors proportional to the number of levels
- IO is generally balanced between random and sequential
- Low CPU overhead
~2000 lines of pure Erlang code in src/*.erl

Hanoi is developed by Trifork, a Riak expert solutions provider. You're most welcome to contact us if you want help optimizing your Riak setup.

Configuration options

Put these values in your app.config in the hanoi section

 {hanoi, [
          {data_root, "./data/hanoi"},
          {compress, none | snappy | gzip},
          {sync_strategy, none | sync | {seconds, N}},
          {page_size, 8192},
          {write_buffer_size, 524288},  % 512kB
          {read_buffer_size, 524288},  % 512kB

          %%
          %% The merge strategy is one of `fast' or `predictable'.
          %% Both have same log2(N) worst case, but `fast' is
          %% sometimes faster; yielding latency fluctuations.
          %%
          {merge_strategy, fast | predictable}
         ]},

How to deploy Hanoi as a Riak/KV backend

This storage engine can function as an alternative backend for Basho's Riak/KV.

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

git clone git://github.com/basho/riak.git
cd riak/deps
git clone git://github.com/basho/hanoi.git
cd ..
./deps/hanoi/enable-hanoi