greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity
HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.
217 commits 4 branches 0 tags 2.4 MiB
Erlang 94.9%
Shell 3.5%
Nix 1.1%
Makefile 0.5%
Find a file
Kresten Krab Thorup 01ea88b67c Implement hibernation for readers too 
This enables all open files in a merge worker
to be closed while it is waiting for work to do.
2012-05-01 02:12:02 +02:00
doc sample 20 min run (MacBook Air/SSD) 2012-04-26 01:51:08 +02:00
include Introduce plain_rpc 2012-04-26 00:49:57 +02:00
src Implement hibernation for readers too 2012-05-01 02:12:02 +02:00
test Tree writing code was broken 2012-04-28 18:35:35 +02:00
.gitignore Provide a Makefile for running dialyzer. 2012-01-06 21:22:23 +01:00
DESIGN.md Cleanup 2012-04-24 10:37:45 -04:00
enable-hanoi Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
LICENSE * Changed "lookup" to "get" just because 2012-04-15 10:35:39 -04:00
Makefile Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
README.md New config: {read|write}_buffer_size 2012-04-30 00:06:42 +02:00
rebar Initial work-in-progress 2012-01-04 15:05:31 +01:00
rebar.config Add riak_core dependency 2012-04-28 18:45:53 +02:00
TODO Update README/TODO 2012-04-28 18:42:04 +02:00
visualize-hanoi.sh Refactor merge work computation 2012-04-30 23:34:27 +02:00

README.md

Hanoi Ordered Key/Value Storage

Hanoi implements an ordered key/value storage engine, implemented using "doubling sizes" persistent ordered sets of key/value pairs, much like LevelDB.

Here's the bullet list:

  • Insert, Delete and Read all have worst case log2(N) latency.
  • Incremental space reclaimation: The cost of evicting stale key/values is amortized into insertion
    • you don't need a separate eviction thread to keep memory use low
    • you don't need to schedule merges to happen at off-peak hours
  • Operations-friendly "append-only" storage
    • allows you to backup live system
    • crash-recovery is very fast and the logic is straight forward
    • All data subject to CRC32 checksums
  • Supports efficient range queries
    • Riak secondary indexing
    • Fast key and bucket listing
  • Uses bloom filters to avoid unnecessary lookups on disk
  • Efficient resource utilization
    • Doesn't store all keys in memory
    • Uses a modest number of file descriptors proportional to the number of levels
    • IO is generally balanced between random and sequential
    • Low CPU overhead
  • ~2000 lines of pure Erlang code in src/*.erl

Hanoi is developed by Trifork, a Riak expert solutions provider. You're most welcome to contact us if you want help optimizing your Riak setup.

Configuration options

Put these values in your app.config in the hanoi section

 {hanoi, [
          {data_root, "./data/hanoi"},
          {compress, none | snappy | gzip},
          {sync_strategy, none | sync | {seconds, N}},
          {page_size, 8192}
          {write_buffer_size, 524288}  % 512kB
          {read_buffer_size, 524288}  % 512kB
         ]},

How to deploy Hanoi as a Riak/KV backend

This storage engine can function as an alternative backend for Basho's Riak/KV.

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

  1. git clone git://github.com/basho/riak.git
  2. cd riak/deps
  3. git clone git://github.com/basho/hanoi.git
  4. cd ..
  5. ./deps/hanoi/enable-hanoi