greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity
HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.
165 commits 4 branches 0 tags 2.4 MiB
Erlang 94.9%
Shell 3.5%
Nix 1.1%
Makefile 0.5%
Find a file
Kresten Krab Thorup 8b725cceaa Improve recovery 
This improves recovery two-fold:

1. make sure that we actually wait for initial
   merge to complete (issue incremental_merge(0))

2. compute minimum required merge work for merge
   to establish invariant that there's room
   for a new nursery inject any time.
2012-04-23 13:53:42 +02:00
doc Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
include Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
src Improve recovery 2012-04-23 13:53:42 +02:00
test Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
.gitignore Provide a Makefile for running dialyzer. 2012-01-06 21:22:23 +01:00
DESIGN.md Implementation notes belong in "DESIGN.md", not "README.md". 2012-04-22 09:27:17 -04:00
enable-hanoi Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
LICENSE * Changed "lookup" to "get" just because 2012-04-15 10:35:39 -04:00
Makefile Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
README.md Implementation notes belong in "DESIGN.md", not "README.md". 2012-04-22 09:27:17 -04:00
rebar Initial work-in-progress 2012-01-04 15:05:31 +01:00
rebar.config Implement compression + block size 2012-04-23 03:49:08 +02:00
TODO Implement compression + block size 2012-04-23 03:49:08 +02:00
visualize-hanoi.sh Switch order ABCX -> CBAX in visualizer 2012-04-22 23:53:31 +02:00

README.md

Hanoi Key/Value Storage Engine

This Erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees, see docs/10.1.1.44.2782.pdf). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees indicates that this storage engine is quite different in several respects.

The storage engine can function as an alternative backend for Basho's Riak/KV.

Here's the bullet list:

  • Very fast writes and deletes,
  • Reasonably fast reads (N records are stored in log2(N) B-trees),
  • Operations-friendly "append-only" storage (allows you to backup live system, and crash-recovery is very simple)
  • The cost of evicting stale key/values is amortized into insertion, so you don't need to schedule merge to happen at off-peak hours.
  • Supports range queries (and thus eventually Riak 2i.)
  • Doesn't need a boat load of RAM
  • All in 1000 lines of pure Erlang code

Once we're a bit more stable, we'll provide a Riak backend.

Deploying the hanoi for testing with Riak/KV

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

  1. git clone git://github.com/basho/riak.git
  2. cd riak/deps
  3. git clone git://github.com/basho/hanoi.git
  4. cd ..
  5. ./deps/hanoi/enable-hanoi # which does make all devrel