greg/hanoidb

Fork 0

HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.

Find a file

Kresten Krab Thorup 0009e17d4f Change delegate work computation We were delegating too much work. The original algorithm description said that for each insert, "1" unit of merge work has to be done at each level … implying that if nothing needs doing at a level, that "not done work" does not add to work done elsewhere. This fix gets us back to that situation (by always subtracting at least 2^TOP_LEVEL from the presented work amount), while maintaining the (beneficial) effect of chunking merge work at at anything but the last level. Effectively, this reduces the maximum amount of merge work done, also reducing our worst case latency. Now that we understand this, we can refactor the algorithm to delegate "DoneWork", because then each level can determine the total work, and see if any work is left "for me". That's next.		2012-04-30 21:38:53 +02:00
doc	sample 20 min run (MacBook Air/SSD)	2012-04-26 01:51:08 +02:00
include	Introduce plain_rpc	2012-04-26 00:49:57 +02:00
src	Change delegate work computation	2012-04-30 21:38:53 +02:00
test	Tree writing code was broken	2012-04-28 18:35:35 +02:00
.gitignore	Provide a Makefile for running dialyzer.	2012-01-06 21:22:23 +01:00
DESIGN.md	Cleanup	2012-04-24 10:37:45 -04:00
enable-hanoi	Rename "lsm-btree" to "hanoi".	2012-04-21 15:20:39 -04:00
LICENSE	* Changed "lookup" to "get" just because	2012-04-15 10:35:39 -04:00
Makefile	Rename "lsm-btree" to "hanoi".	2012-04-21 15:20:39 -04:00
README.md	New config: {read\|write}_buffer_size	2012-04-30 00:06:42 +02:00
rebar	Initial work-in-progress	2012-01-04 15:05:31 +01:00
rebar.config	Add riak_core dependency	2012-04-28 18:45:53 +02:00
TODO	Update README/TODO	2012-04-28 18:42:04 +02:00
visualize-hanoi.sh	New visualizer	2012-04-24 01:43:53 +02:00

README.md

Hanoi Ordered Key/Value Storage

Hanoi implements an ordered key/value storage engine, implemented using "doubling sizes" persistent ordered sets of key/value pairs, much like LevelDB.

Here's the bullet list:

Insert, Delete and Read all have worst case log₂(N) latency.
Incremental space reclaimation: The cost of evicting stale key/values is amortized into insertion
- you don't need a separate eviction thread to keep memory use low
- you don't need to schedule merges to happen at off-peak hours
Operations-friendly "append-only" storage
- allows you to backup live system
- crash-recovery is very fast and the logic is straight forward
- All data subject to CRC32 checksums
Supports efficient range queries
- Riak secondary indexing
- Fast key and bucket listing
Uses bloom filters to avoid unnecessary lookups on disk
Efficient resource utilization
- Doesn't store all keys in memory
- Uses a modest number of file descriptors proportional to the number of levels
- IO is generally balanced between random and sequential
- Low CPU overhead
~2000 lines of pure Erlang code in src/*.erl

Hanoi is developed by Trifork, a Riak expert solutions provider. You're most welcome to contact us if you want help optimizing your Riak setup.

Configuration options

Put these values in your app.config in the hanoi section

 {hanoi, [
          {data_root, "./data/hanoi"},
          {compress, none | snappy | gzip},
          {sync_strategy, none | sync | {seconds, N}},
          {page_size, 8192}
          {write_buffer_size, 524288}  % 512kB
          {read_buffer_size, 524288}  % 512kB
         ]},

How to deploy Hanoi as a Riak/KV backend

This storage engine can function as an alternative backend for Basho's Riak/KV.

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

git clone git://github.com/basho/riak.git
cd riak/deps
git clone git://github.com/basho/hanoi.git
cd ..
./deps/hanoi/enable-hanoi