greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity
HanoiDB implements an ordered k/v pairs storage engine in Erlang. The primary index is a log-structured merge tree (LSM-BTree) implemented using "doubling sizes" persistent ordered sets of kvp.
168 commits 4 branches 0 tags 2.4 MiB
Erlang 94.9%
Shell 3.5%
Nix 1.1%
Makefile 0.5%
Find a file
Kresten Krab Thorup ff36e401b7 Incremental merge refactor, step #2 
Now incremental merge has a new strategy.
In stead of doing the same amount of merge
work at all levels, we now compute the total
merge work load, and do as much as possible
on the first level, subtract work done, and
delegate to the next level, etc.

The effect of this is that we do more IO on
fewer files, improving sequential-ness of
the workload involved in the incremental merge.
2012-04-24 00:31:28 +02:00
doc Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
include Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
src Incremental merge refactor, step #2 2012-04-24 00:31:28 +02:00
test Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
.gitignore Provide a Makefile for running dialyzer. 2012-01-06 21:22:23 +01:00
DESIGN.md Implementation notes belong in "DESIGN.md", not "README.md". 2012-04-22 09:27:17 -04:00
enable-hanoi Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
LICENSE * Changed "lookup" to "get" just because 2012-04-15 10:35:39 -04:00
Makefile Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
README.md Implementation notes belong in "DESIGN.md", not "README.md". 2012-04-22 09:27:17 -04:00
rebar Initial work-in-progress 2012-01-04 15:05:31 +01:00
rebar.config Implement compression + block size 2012-04-23 03:49:08 +02:00
TODO Implement compression + block size 2012-04-23 03:49:08 +02:00
visualize-hanoi.sh Switch order ABCX -> CBAX in visualizer 2012-04-22 23:53:31 +02:00

README.md

Hanoi Key/Value Storage Engine

This Erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees, see docs/10.1.1.44.2782.pdf). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees indicates that this storage engine is quite different in several respects.

The storage engine can function as an alternative backend for Basho's Riak/KV.

Here's the bullet list:

  • Very fast writes and deletes,
  • Reasonably fast reads (N records are stored in log2(N) B-trees),
  • Operations-friendly "append-only" storage (allows you to backup live system, and crash-recovery is very simple)
  • The cost of evicting stale key/values is amortized into insertion, so you don't need to schedule merge to happen at off-peak hours.
  • Supports range queries (and thus eventually Riak 2i.)
  • Doesn't need a boat load of RAM
  • All in 1000 lines of pure Erlang code

Once we're a bit more stable, we'll provide a Riak backend.

Deploying the hanoi for testing with Riak/KV

You can deploy hanoi into a Riak devrel cluster using the enable-hanoi script. Clone the riak repo, change your working directory to it, and then execute the enable-hanoi script. It adds hanoi as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the hanoi storage backend.

  1. git clone git://github.com/basho/riak.git
  2. cd riak/deps
  3. git clone git://github.com/basho/hanoi.git
  4. cd ..
  5. ./deps/hanoi/enable-hanoi # which does make all devrel