greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity
HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.
215 commits 4 branches 0 tags 2.4 MiB
Erlang 94.9%
Shell 3.5%
Nix 1.1%
Makefile 0.5%
Find a file
Kresten Krab Thorup c8e403af8c Refactor merge work computation 
Now merge work computation is close to ideal.
It does not take into account the actual size
of files at each level, but we have not figured
out how to utilize that knowledge.
2012-04-30 23:34:27 +02:00
doc sample 20 min run (MacBook Air/SSD) 2012-04-26 01:51:08 +02:00
include Introduce plain_rpc 2012-04-26 00:49:57 +02:00
src Refactor merge work computation 2012-04-30 23:34:27 +02:00
test Tree writing code was broken 2012-04-28 18:35:35 +02:00
.gitignore Provide a Makefile for running dialyzer. 2012-01-06 21:22:23 +01:00
DESIGN.md Cleanup 2012-04-24 10:37:45 -04:00
enable-hanoi Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
LICENSE * Changed "lookup" to "get" just because 2012-04-15 10:35:39 -04:00
Makefile Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
README.md New config: {read|write}_buffer_size 2012-04-30 00:06:42 +02:00
rebar Initial work-in-progress 2012-01-04 15:05:31 +01:00
rebar.config Add riak_core dependency 2012-04-28 18:45:53 +02:00
TODO Update README/TODO 2012-04-28 18:42:04 +02:00
visualize-hanoi.sh Refactor merge work computation 2012-04-30 23:34:27 +02:00

README.md

Hanoi Ordered Key/Value Storage

Hanoi implements an ordered key/value storage engine, implemented using "doubling sizes" persistent ordered sets of key/value pairs, much like LevelDB.

Here's the bullet list:

  • Insert, Delete and Read all have worst case log2(N) latency.
  • Incremental space reclaimation: The cost of evicting stale key/values is amortized into insertion
    • you don't need a separate eviction thread to keep memory use low
    • you don't need to schedule merges to happen at off-peak hours
  • Operations-friendly "append-only" storage
    • allows you to backup live system
    • crash-recovery is very fast and the logic is straight forward
    • All data subject to CRC32 checksums
  • Supports efficient range queries
    • Riak secondary indexing
    • Fast key and bucket listing
  • Uses bloom filters to avoid unnecessary lookups on disk
  • Efficient resource utilization
    • Doesn't store all keys in memory
    • Uses a modest number of file descriptors proportional to the number of levels
    • IO is generally balanced between random and sequential
    • Low CPU overhead
  • ~2000 lines of pure Erlang code in src/*.erl

Hanoi is developed by Trifork, a Riak expert solutions provider. You're most welcome to contact us if you want help optimizing your Riak setup.

Configuration options

Put these values in your app.config in the hanoi section

 {hanoi, [
          {data_root, "./data/hanoi"},
          {compress, none | snappy | gzip},
          {sync_strategy, none | sync | {seconds, N}},
          {page_size, 8192}
          {write_buffer_size, 524288}  % 512kB
          {read_buffer_size, 524288}  % 512kB
         ]},

How to deploy Hanoi as a Riak/KV backend

This storage engine can function as an alternative backend for Basho's Riak/KV.

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

  1. git clone git://github.com/basho/riak.git
  2. cd riak/deps
  3. git clone git://github.com/basho/hanoi.git
  4. cd ..
  5. ./deps/hanoi/enable-hanoi