Merge pull request #3 from basho/rename-to-hanoi

Rename "lsm_btree" to "hanoi".
This commit is contained in:
Gregory Burd 2012-04-21 12:30:07 -07:00
commit 638f2d56ee
27 changed files with 267 additions and 267 deletions

View file

@ -28,7 +28,7 @@ clean-test-btrees:
rm -fr .eunit/Btree_* .eunit/simple
plt: compile
$(DIALYZER) --build_plt --output_plt .lsm_btree.plt \
$(DIALYZER) --build_plt --output_plt .hanoi.plt \
-pa deps/plain_fsm/ebin \
-pa deps/ebloom/ebin \
deps/plain_fsm/ebin \
@ -36,7 +36,7 @@ plt: compile
--apps kernel stdlib
analyze: compile
$(DIALYZER) --plt .lsm_btree.plt \
$(DIALYZER) --plt .hanoi.plt \
-pa deps/plain_fsm/ebin \
-pa deps/ebloom/ebin \
ebin

View file

@ -1,22 +1,22 @@
# LSM B-Tree Storage
# Hanoi Key/Value Storage Engine
This erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees and Fractal Trees indicates that this storage engine is quite different in several respects.
This Erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees, see docs/10.1.1.44.2782.pdf). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees indicates that this storage engine is quite different in several respects.
The storage engine may eventually provide an alternative backend for Basho's Riak. Of the existing backends, `lsm_btree` is closest to LevelDB in operational characteristics, but it is implemented in just ~1000 lines of Erlang.
The storage engine can function as an alternative backend for Basho's Riak/KV.
Here's the bullet list:
- Very fast writes and deletes,
- Reasonably fast reads (N records are stored in log<sub>2</sub>(N) B-trees),
- Operations-friendly "append-only" storage (allows you to backup live system, and crash-recovery is very simple)
- The cost of evicting stale key/values is amortized into insertion, so you don't need to schedule merge to happen at off-peak hours.
- The cost of evicting stale key/values is amortized into insertion, so you don't need to schedule merge to happen at off-peak hours.
- Supports range queries (and thus eventually Riak 2i.)
- Doesn't need a boat load of RAM
- All in 1000 lines of pure Erlang code
Once we're a bit more stable, we'll provide a Riak backend.
## How a LSM-BTree Works
## How this LSM-BTree Works
If there are N records, there are in log<sub>2</sub>(N) levels (each being a plain B-tree in a file named "A-*level*.data"). The file `A-0.data` has 1 record, `A-1.data` has 2 records, `A-2.data` has 4 records, and so on: `A-n.data` has 2<sup>n</sup> records.
@ -30,11 +30,11 @@ Lookup is quite simple: starting at `A-0.data`, the sought for Key is searched i
### Insertion
Insertion works by a mechanism known as B-tree injection. Insertion always starts by constructing a fresh B-tree with 1 element in it, and "injecting" that B-tree into level #0. So you always inject a B-tree of the same size as the size of the level you're injecting it into.
- If the level being injected into empty (there is no A-*level*.data file), then the injected B-tree becomes the contents for that level (we just rename the file).
- Otherwise,
- If the level being injected into empty (there is no A-*level*.data file), then the injected B-tree becomes the contents for that level (we just rename the file).
- Otherwise,
- The injected tree file is renamed to B-*level*.data;
- The files A-*level*.data and B-*level*.data are merged into a new temporary B-tree (of roughly double size), X-*level*.data.
- The outcome of the merge is then injected into the next level.
- The files A-*level*.data and B-*level*.data are merged into a new temporary B-tree (of roughly double size), X-*level*.data.
- The outcome of the merge is then injected into the next level.
While merging, lookups at level *n* first consults the B-*n*.data file, then the A-*n*.data file. At a given level, there can only be one merge operation active.
@ -48,23 +48,23 @@ Deletes are the same: they are also done by inserting a tombstone (a special val
The really clever thing about this storage mechanism is that merging is guaranteed to be able to "keep up" with insertion. Bitcask for instance has a similar merging phase, but it is separated from insertion. This means that there can suddenly be a lot of catching up to do. The flip side is that you can then decide to do all merging at off-peak hours, but it is yet another thing that need to be configured.
With LSM B-Trees; back-pressure is provided by the injection mechanism, which only returns when an injection is complete. Thus, every 2nd insert needs to wait for level #0 to finish the required merging; which - assuming merging has linear I/O complexity - is enough to guarantee that the merge mechanism can keep up at higher-numbered levels.
With LSM B-Trees; back-pressure is provided by the injection mechanism, which only returns when an injection is complete. Thus, every 2nd insert needs to wait for level #0 to finish the required merging; which - assuming merging has linear I/O complexity - is enough to guarantee that the merge mechanism can keep up at higher-numbered levels.
A further trouble is that merging does in fact not have completely linear I/O complexity, because reading from a small file that was recently written is faster that reading from a file that was written a long time ago (because of OS-level caching); thus doing a merge at level #*N+1* is sometimes more than twice as slow as doing a merge at level #*N*. Because of this, sustained insert pressure may produce a situation where the system blocks while merging, though it does require an extremely high level of inserts. We're considering ways to alleviate this.
Merging can be going on concurrently at each level (in preparation for an injection to the next level), which lets you utilize available multi-core capacity to merge.
Merging can be going on concurrently at each level (in preparation for an injection to the next level), which lets you utilize available multi-core capacity to merge.
### Deploying the lsm_btree for testing with Riak/KV
### Deploying the hanoi for testing with Riak/KV
You can deploy `lsm_btree` into a Riak devrel cluster using the
`enable-lsm_btree` script. Clone the `riak` repo, change your working directory
to it, and then execute the `enable-lsm_btree` script. It adds `lsm_btree` as a
You can deploy `hanoi` into a Riak devrel cluster using the
`enable-hanoi` script. Clone the `riak` repo, change your working directory
to it, and then execute the `enable-hanoi` script. It adds `hanoi` as a
dependency, runs `make all devrel`, and then modifies the configuration
settings of the resulting dev nodes to use the lsm_btree storage backend.
settings of the resulting dev nodes to use the hanoi storage backend.
1. `git clone git://github.com/basho/riak.git`
1. `cd riak/deps`
1. `git clone git://github.com/basho/lsm_btree.git`
1. `git clone git://github.com/basho/hanoi.git`
1. `cd ..`
1. `./deps/lsm_btree/enable-lsm_btree` # which does `make all devrel`
1. `./deps/hanoi/enable-hanoi` # which does `make all devrel`

12
TODO
View file

@ -1,4 +1,4 @@
* lsm_btree
* hanoi
* [cleanup] add @doc strings and and -spec's
* [cleanup] check to make sure every error returns with a reason {error, Reason}
* [feature] statistics
@ -23,7 +23,7 @@
serviced, then picks up from there again. So you could see intermittent
puts in a subsequent batch of results.
* riak_kv_lsm_tree_backend
* riak_kv_hanoie_backend
* add support for time-based expiry
* finish support for 2i
* add stats collection
@ -31,16 +31,16 @@
PHASE 2:
* lsm_btree
* hanoi
* Define a standard struct which is the metadata added at the end of the
file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written
in lsm_btree_writer:flush_nodes, and read in lsm_btree_reader:open2.
in hanoi_writer:flush_nodes, and read in hanoi_reader:open2.
* [feature] compression, encryption on disk
PHASE 3:
* lsm_ixdb
* lsm_{btree, ctrie, ...} support for sub-databases and associations with
* hanoi{btree, trie, ...} support for sub-databases and associations with
different index types
* [major change] add more CAPABILITIES such as
test-and-set(Fun, Key, Value) - to compare a vclock quickly, to speed up
@ -56,7 +56,7 @@ REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS:
* http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.44.2782&rep=rep1&type=pdf
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in lsm_btree.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in hanoi.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads.

View file

Before

Width:  |  Height:  |  Size: 119 KiB

After

Width:  |  Height:  |  Size: 119 KiB

View file

@ -1,12 +1,12 @@
#!/bin/sh
# This script adds lsm_btree to a riak github repo. Run it in the riak repo
# This script adds hanoi to a riak github repo. Run it in the riak repo
# directory.
#
# First it adds lsm_btree, then runs "make all devrel" and then enables the
# lsm_btree storage backend in the resulting dev nodes.
# First it adds hanoi, then runs "make all devrel" and then enables the
# hanoi storage backend in the resulting dev nodes.
#
# This script is intended to be temporary. Once lsm_btree is made into a proper
# This script is intended to be temporary. Once hanoi is made into a proper
# riak citizen, this script will no longer be needed.
set -e
@ -35,17 +35,17 @@ fi
./rebar get-deps
file=./deps/riak_kv/src/riak_kv.app.src
if ! grep -q lsm_btree $file ; then
if ! grep -q hanoi $file ; then
echo
echo "Modifying $file, saving the original as ${file}.orig ..."
perl -i.orig -pe '/\bos_mon,/ && print qq( lsm_btree,\n)' $file
perl -i.orig -pe '/\bos_mon,/ && print qq( hanoi,\n)' $file
fi
file=./deps/riak_kv/rebar.config
if ! grep -q lsm_btree $file ; then
if ! grep -q hanoi $file ; then
echo
echo "Modifying $file, saving the original as ${file}.orig ..."
perl -i.orig -pe '/\bsext\b/ && print qq( {lsm_btree, ".*", {git, "git\@github.com:basho/lsm_btree.git", "master"}},\n)' $file
perl -i.orig -pe '/\bsext\b/ && print qq( {hanoi, ".*", {git, "git\@github.com:basho/hanoi.git", "master"}},\n)' $file
fi
./rebar get-deps
@ -55,6 +55,6 @@ make all devrel
echo
echo 'Modifying all dev/dev*/etc/app.config files, saving originals with .orig suffix...'
perl -i.orig -ne 'if (/\bstorage_backend,/) { s/(storage_backend, )[^\}]+/\1riak_kv_lsm_btree_backend/; print } elsif (/\{eleveldb,/) { $eleveldb++; print } elsif ($eleveldb && /^\s+\]\},/) { $eleveldb = 0; print; print qq(\n {lsm_btree, [\n {data_root, "./data/lsm_btree"}\n ]},\n\n) } else { print }' dev/dev*/etc/app.config
perl -i.orig -ne 'if (/\bstorage_backend,/) { s/(storage_backend, )[^\}]+/\1riak_kv_hanoi_backend/; print } elsif (/\{eleveldb,/) { $eleveldb++; print } elsif ($eleveldb && /^\s+\]\},/) { $eleveldb = 0; print; print qq(\n {hanoi, [\n {data_root, "./data/hanoi"}\n ]},\n\n) } else { print }' dev/dev*/etc/app.config
exit 0

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(basho_bench_driver_lsm_btree).
-module(basho_bench_driver_hanoi).
-record(state, { tree,
filename,
@ -33,7 +33,7 @@
-export([new/1,
run/4]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-include_lib("basho_bench/include/basho_bench.hrl").
-record(btree_range, { from_key = <<>> :: binary(),
@ -48,20 +48,20 @@
new(_Id) ->
%% Make sure bitcask is available
case code:which(lsm_btree) of
case code:which(hanoi) of
non_existing ->
?FAIL_MSG("~s requires lsm_btree to be available on code path.\n",
?FAIL_MSG("~s requires hanoi to be available on code path.\n",
[?MODULE]);
_ ->
ok
end,
%% Get the target directory
Dir = basho_bench_config:get(lsm_btree_dir, "."),
Filename = filename:join(Dir, "test.lsm_btree"),
Dir = basho_bench_config:get(hanoi_dir, "."),
Filename = filename:join(Dir, "test.hanoi"),
%% Look for sync interval config
case basho_bench_config:get(lsm_btree_sync_interval, infinity) of
case basho_bench_config:get(hanoi_sync_interval, infinity) of
Value when is_integer(Value) ->
SyncInterval = Value;
infinity ->
@ -69,9 +69,9 @@ new(_Id) ->
end,
%% Get any bitcask flags
case lsm_btree:open(Filename) of
case hanoi:open(Filename) of
{error, Reason} ->
?FAIL_MSG("Failed to open lsm btree in ~s: ~p\n", [Filename, Reason]);
?FAIL_MSG("Failed to open hanoi in ~s: ~p\n", [Filename, Reason]);
{ok, FBTree} ->
{ok, #state { tree = FBTree,
filename = Filename,
@ -80,7 +80,7 @@ new(_Id) ->
end.
run(get, KeyGen, _ValueGen, State) ->
case lsm_btree:lookup(State#state.tree, KeyGen()) of
case hanoi:lookup(State#state.tree, KeyGen()) of
{ok, _Value} ->
{ok, State};
not_found ->
@ -89,14 +89,14 @@ run(get, KeyGen, _ValueGen, State) ->
{error, Reason}
end;
run(put, KeyGen, ValueGen, State) ->
case lsm_btree:put(State#state.tree, KeyGen(), ValueGen()) of
case hanoi:put(State#state.tree, KeyGen(), ValueGen()) of
ok ->
{ok, State};
{error, Reason} ->
{error, Reason}
end;
run(delete, KeyGen, _ValueGen, State) ->
case lsm_btree:delete(State#state.tree, KeyGen()) of
case hanoi:delete(State#state.tree, KeyGen()) of
ok ->
{ok, State};
{error, Reason} ->
@ -105,7 +105,7 @@ run(delete, KeyGen, _ValueGen, State) ->
run(fold_100, KeyGen, _ValueGen, State) ->
[From,To] = lists:usort([KeyGen(), KeyGen()]),
case lsm_btree:sync_fold_range(State#state.tree,
case hanoi:sync_fold_range(State#state.tree,
fun(_Key,_Value,Count) ->
Count+1
end,

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
{application, lsm_btree,
{application, hanoi,
[
{description, ""},
{vsn, "1.0.0"},
@ -31,6 +31,6 @@
kernel,
stdlib
]},
{mod, {lsm_btree_app, []}},
{mod, {hanoi_app, []}},
{env, []}
]}.

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree).
-module(hanoi).
-author('Kresten Krab Thorup <krab@trifork.com>').
@ -34,9 +34,9 @@
-export([open/1, close/1, get/2, lookup/2, delete/2, put/3,
async_range/2, async_fold_range/4, sync_range/2, sync_fold_range/4]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-include_lib("kernel/include/file.hrl").
-include_lib("include/lsm_btree.hrl").
-include_lib("include/hanoi.hrl").
-record(state, { top, nursery, dir }).
@ -87,7 +87,7 @@ sync_receive_fold_range(MRef, PID,Fun,Acc0) ->
{ok, Fun(K,V,Acc0)}
catch
Class:Exception ->
lager:warning("Exception in lsm_btree fold: ~p", [Exception]),
lager:warning("Exception in hanoi fold: ~p", [Exception]),
{'EXIT', Class, Exception, erlang:get_stacktrace()}
end
of
@ -106,7 +106,7 @@ sync_receive_fold_range(MRef, PID,Fun,Acc0) ->
{ok, kvfoldl(Fun,Acc0,KVs)}
catch
Class:Exception ->
lager:warning("Exception in lsm_btree fold: ~p", [Exception]),
lager:warning("Exception in hanoi fold: ~p", [Exception]),
{'EXIT', Class, Exception, erlang:get_stacktrace()}
end
of
@ -180,12 +180,12 @@ init([Dir]) ->
case file:read_file_info(Dir) of
{ok, #file_info{ type=directory }} ->
{ok, TopLevel} = open_levels(Dir),
{ok, Nursery} = lsm_btree_nursery:recover(Dir, TopLevel);
{ok, Nursery} = hanoi_nursery:recover(Dir, TopLevel);
{error, E} when E =:= enoent ->
ok = file:make_dir(Dir),
{ok, TopLevel} = lsm_btree_level:open(Dir, ?TOP_LEVEL, undefined),
{ok, Nursery} = lsm_btree_nursery:new(Dir)
{ok, TopLevel} = hanoi_level:open(Dir, ?TOP_LEVEL, undefined),
{ok, Nursery} = hanoi_nursery:new(Dir)
end,
{ok, #state{ top=TopLevel, dir=Dir, nursery=Nursery }}.
@ -216,7 +216,7 @@ open_levels(Dir) ->
TopLevel =
lists:foldl( fun(LevelNo, Prev) ->
{ok, Level} = lsm_btree_level:open(Dir,LevelNo,Prev),
{ok, Level} = hanoi_level:open(Dir,LevelNo,Prev),
Level
end,
undefined,
@ -256,15 +256,15 @@ code_change(_OldVsn, State, _Extra) ->
handle_call({async_range, Sender, Range}, _From, State=#state{ top=TopLevel, nursery=Nursery }) ->
{ok, FoldWorkerPID} = lsm_btree_fold_worker:start(Sender),
lsm_btree_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = lsm_btree_level:async_range(TopLevel, FoldWorkerPID, Range),
{ok, FoldWorkerPID} = hanoi_fold_worker:start(Sender),
hanoi_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = hanoi_level:async_range(TopLevel, FoldWorkerPID, Range),
{reply, Result, State};
handle_call({sync_range, Sender, Range}, _From, State=#state{ top=TopLevel, nursery=Nursery }) ->
{ok, FoldWorkerPID} = lsm_btree_fold_worker:start(Sender),
lsm_btree_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = lsm_btree_level:sync_range(TopLevel, FoldWorkerPID, Range),
{ok, FoldWorkerPID} = hanoi_fold_worker:start(Sender),
hanoi_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = hanoi_level:sync_range(TopLevel, FoldWorkerPID, Range),
{reply, Result, State};
handle_call({put, Key, Value}, _From, State) when is_binary(Key), is_binary(Value) ->
@ -276,20 +276,20 @@ handle_call({delete, Key}, _From, State) when is_binary(Key) ->
{reply, ok, State2};
handle_call({get, Key}, _From, State=#state{ top=Top, nursery=Nursery } ) when is_binary(Key) ->
case lsm_btree_nursery:lookup(Key, Nursery) of
case hanoi_nursery:lookup(Key, Nursery) of
{value, ?TOMBSTONE} ->
{reply, not_found, State};
{value, Value} when is_binary(Value) ->
{reply, {ok, Value}, State};
none ->
Reply = lsm_btree_level:lookup(Top, Key),
Reply = hanoi_level:lookup(Top, Key),
{reply, Reply, State}
end;
handle_call(close, _From, State=#state{top=Top}) ->
try
{ok, State2} = flush_nursery(State),
ok = lsm_btree_level:close(Top),
ok = hanoi_level:close(Top),
{stop, normal, ok, State2}
catch
E:R ->
@ -298,10 +298,10 @@ handle_call(close, _From, State=#state{top=Top}) ->
end.
do_put(Key, Value, State=#state{ nursery=Nursery, top=Top }) ->
{ok, Nursery2} = lsm_btree_nursery:add_maybe_flush(Key, Value, Nursery, Top),
{ok, Nursery2} = hanoi_nursery:add_maybe_flush(Key, Value, Nursery, Top),
{ok, State#state{ nursery=Nursery2 }}.
flush_nursery(State=#state{nursery=Nursery, top=Top, dir=Dir}) ->
ok = lsm_btree_nursery:finish(Nursery, Top),
{ok, Nursery2} = lsm_btree_nursery:new(Dir),
ok = hanoi_nursery:finish(Nursery, Top),
{ok, Nursery2} = hanoi_nursery:new(Dir),
{ok, State#state{ nursery=Nursery2 }}.

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_app).
-module(hanoi_app).
-author('Kresten Krab Thorup <krab@trifork.com>').
-behaviour(application).
@ -35,7 +35,7 @@
%% ===================================================================
start(_StartType, _StartArgs) ->
lsm_btree_sup:start_link().
hanoi_sup:start_link().
stop(_State) ->
ok.

View file

@ -5,7 +5,7 @@
{concurrent, 1}.
{driver, basho_bench_driver_lsm_btree}.
{driver, basho_bench_driver_hanoi}.
{key_generator, {int_to_bin,{uniform_int, 5000000}}}.
@ -14,13 +14,13 @@
{operations, [{get, 1}, {put, 1}]}.
%% the second element in the list below (e.g., "../../public/bitcask") must point to
%% the relevant directory of a lsm_btree installation
%% the relevant directory of a hanoi installation
{code_paths, ["deps/stats",
"../lsm_btree/ebin",
"../lsm_btree/deps/plain_fsm/ebin",
"../lsm_btree/deps/ebloom/ebin"
"../hanoi/ebin",
"../hanoi/deps/plain_fsm/ebin",
"../hanoi/deps/ebloom/ebin"
]}.
{bitcask_dir, "/tmp/lsm_btree.bench"}.
{bitcask_dir, "/tmp/hanoi.bench"}.
{bitcask_flags, [o_sync]}.

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_fold_worker).
-module(hanoi_fold_worker).
-author('Kresten Krab Thorup <krab@trifork.com>').
@ -59,7 +59,7 @@
-behavior(plain_fsm).
-export([data_vsn/0, code_change/3]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-record(state, {sendto}).

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,14 +22,14 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_level).
-module(hanoi_level).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include("include/lsm_btree.hrl").
-include("src/lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("src/hanoi.hrl").
%%
%% Manages a "pair" of lsm_index (or rathern, 0, 1 or 2), and governs
%% Manages a "pair" of hanoi index (or rathern, 0, 1 or 2), and governs
%% the process of injecting/merging parent trees into this pair.
%%
@ -162,12 +162,12 @@ initialize2(State) ->
file:delete(BFileName),
ok = file:rename(MFileName, AFileName),
{ok, BT} = lsm_btree_reader:open(AFileName, random),
{ok, BT} = hanoi_reader:open(AFileName, random),
case file:read_file_info(CFileName) of
{ok, _} ->
file:rename(CFileName, BFileName),
{ok, BT2} = lsm_btree_reader:open(BFileName, random),
{ok, BT2} = hanoi_reader:open(BFileName, random),
check_begin_merge_then_loop(State#state{ a= BT, b=BT2 });
{error, enoent} ->
@ -177,12 +177,12 @@ initialize2(State) ->
{error, enoent} ->
case file:read_file_info(BFileName) of
{ok, _} ->
{ok, BT1} = lsm_btree_reader:open(AFileName, random),
{ok, BT2} = lsm_btree_reader:open(BFileName, random),
{ok, BT1} = hanoi_reader:open(AFileName, random),
{ok, BT2} = hanoi_reader:open(BFileName, random),
case file:read_file_info(CFileName) of
{ok, _} ->
{ok, BT3} = lsm_btree_reader:open(CFileName, random);
{ok, BT3} = hanoi_reader:open(CFileName, random);
{error, enoent} ->
BT3 = undefined
end,
@ -193,7 +193,7 @@ initialize2(State) ->
case file:read_file_info(AFileName) of
{ok, _} ->
{ok, BT1} = lsm_btree_reader:open(AFileName, random),
{ok, BT1} = hanoi_reader:open(AFileName, random),
main_loop(State#state{ a=BT1 });
{error, enoent} ->
@ -236,7 +236,7 @@ main_loop(State = #state{ next=Next }) ->
SetPos = #state.c
end,
ok = file:rename(FileName, ToFileName),
{ok, BT} = lsm_btree_reader:open(ToFileName, random),
{ok, BT} = hanoi_reader:open(ToFileName, random),
reply(From, ok),
check_begin_merge_then_loop(setelement(SetPos, State, BT));
@ -332,7 +332,7 @@ main_loop(State = #state{ next=Next }) ->
%% rpc would fail when we fall off the cliff
if Next == undefined -> ok;
true ->
lsm_btree_level:close(Next)
hanoi_level:close(Next)
end,
ok;
@ -445,7 +445,7 @@ main_loop(State = #state{ next=Next }) ->
% then, rename M to A, and open it
AFileName = filename("A",State2),
ok = file:rename(MFileName, AFileName),
{ok, BT} = lsm_btree_reader:open(AFileName, random),
{ok, BT} = hanoi_reader:open(AFileName, random),
% iff there is a C file, then move it to B position
% TODO: consider recovery for this
@ -535,7 +535,7 @@ do_lookup(_Key, [Pid]) when is_pid(Pid) ->
do_lookup(Key, [undefined|Rest]) ->
do_lookup(Key, Rest);
do_lookup(Key, [BT|Rest]) ->
case lsm_btree_reader:lookup(BT, Key) of
case hanoi_reader:lookup(BT, Key) of
{ok, ?TOMBSTONE} ->
not_found;
{ok, Result} ->
@ -545,7 +545,7 @@ do_lookup(Key, [BT|Rest]) ->
end.
close_if_defined(undefined) -> ok;
close_if_defined(BT) -> lsm_btree_reader:close(BT).
close_if_defined(BT) -> hanoi_reader:close(BT).
stop_if_defined(undefined) -> ok;
stop_if_defined(MergePid) when is_pid(MergePid) ->
@ -567,7 +567,7 @@ begin_merge(State) ->
file:delete(XFileName),
MergePID = proc_lib:spawn_link(fun() ->
{ok, OutCount} = lsm_btree_merger:merge(AFileName, BFileName, XFileName,
{ok, OutCount} = hanoi_merger:merge(AFileName, BFileName, XFileName,
?BTREE_SIZE(State#state.level + 1),
State#state.next =:= undefined),
% error_logger:info_msg("merge done ~p,~p -> ~p~n", [AFileName, BFileName, XFileName]),
@ -582,8 +582,8 @@ close_and_delete_a_and_b(State) ->
AFileName = filename("A",State),
BFileName = filename("B",State),
ok = lsm_btree_reader:close(State#state.a),
ok = lsm_btree_reader:close(State#state.b),
ok = hanoi_reader:close(State#state.a),
ok = hanoi_reader:close(State#state.b),
ok = file:delete(AFileName),
ok = file:delete(BFileName),
@ -600,7 +600,7 @@ start_range_fold(FileName, WorkerPID, Range) ->
PID =
proc_lib:spawn( fun() ->
erlang:link(WorkerPID),
{ok, File} = lsm_btree_reader:open(FileName, sequential),
{ok, File} = hanoi_reader:open(FileName, sequential),
do_range_fold(File, WorkerPID, self(), Range),
erlang:unlink(WorkerPID),
@ -609,12 +609,12 @@ start_range_fold(FileName, WorkerPID, Range) ->
end ),
{ok, PID}.
-spec do_range_fold(BT :: lsm_btree_reader:read_file(),
-spec do_range_fold(BT :: hanoi_reader:read_file(),
WorkerPID :: pid(),
SelfOrRef :: pid() | reference(),
Range :: #btree_range{} ) -> ok.
do_range_fold(BT, WorkerPID, SelfOrRef, Range) ->
case lsm_btree_reader:range_fold(fun(Key,Value,_) ->
case hanoi_reader:range_fold(fun(Key,Value,_) ->
WorkerPID ! {level_result, SelfOrRef, Key, Value},
ok
end,

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_merger).
-module(hanoi_merger).
-author('Kresten Krab Thorup <krab@trifork.com>').
%%
@ -31,7 +31,7 @@
-export([merge/5]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
%%
%% Most likely, there will be plenty of I/O being generated by
@ -41,29 +41,29 @@
-define(LOCAL_WRITER, true).
merge(A,B,C, Size, IsLastLevel) ->
{ok, BT1} = lsm_btree_reader:open(A, sequential),
{ok, BT2} = lsm_btree_reader:open(B, sequential),
{ok, BT1} = hanoi_reader:open(A, sequential),
{ok, BT2} = hanoi_reader:open(B, sequential),
case ?LOCAL_WRITER of
true ->
{ok, Out} = lsm_btree_writer:init([C, Size]);
{ok, Out} = hanoi_writer:init([C, Size]);
false ->
{ok, Out} = lsm_btree_writer:open(C, Size)
{ok, Out} = hanoi_writer:open(C, Size)
end,
{node, AKVs} = lsm_btree_reader:first_node(BT1),
{node, BKVs} = lsm_btree_reader:first_node(BT2),
{node, AKVs} = hanoi_reader:first_node(BT1),
{node, BKVs} = hanoi_reader:first_node(BT2),
{ok, Count, Out2} = scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, 0, {0, none}),
%% finish stream tree
ok = lsm_btree_reader:close(BT1),
ok = lsm_btree_reader:close(BT2),
ok = hanoi_reader:close(BT1),
ok = hanoi_reader:close(BT2),
case ?LOCAL_WRITER of
true ->
{stop, normal, ok, _} = lsm_btree_writer:handle_call(close, self(), Out2);
{stop, normal, ok, _} = hanoi_writer:handle_call(close, self(), Out2);
false ->
ok = lsm_btree_writer:close(Out2)
ok = hanoi_writer:close(Out2)
end,
{ok, Count}.
@ -88,7 +88,7 @@ scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, {0, FromPID}) ->
end;
scan(BT1, BT2, Out, IsLastLevel, [], BKVs, Count, Step) ->
case lsm_btree_reader:next_node(BT1) of
case hanoi_reader:next_node(BT1) of
{node, AKVs} ->
scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, Step);
end_of_data ->
@ -96,7 +96,7 @@ scan(BT1, BT2, Out, IsLastLevel, [], BKVs, Count, Step) ->
end;
scan(BT1, BT2, Out, IsLastLevel, AKVs, [], Count, Step) ->
case lsm_btree_reader:next_node(BT2) of
case hanoi_reader:next_node(BT2) of
{node, BKVs} ->
scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, Step);
end_of_data ->
@ -107,9 +107,9 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
if Key1 < Key2 ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key1, Value1}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key1, Value1}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key1, Value1)
ok = hanoi_writer:add(Out2=Out, Key1, Value1)
end,
scan(BT1, BT2, Out2, IsLastLevel, AT, BKVs, Count+1, step(Step));
@ -117,9 +117,9 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
Key2 < Key1 ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key2, Value2}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key2, Value2}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key2, Value2)
ok = hanoi_writer:add(Out2=Out, Key2, Value2)
end,
scan(BT1, BT2, Out2, IsLastLevel, AKVs, BT, Count+1, step(Step));
@ -129,15 +129,15 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
true ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key2, Value2}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key2, Value2}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key2, Value2)
ok = hanoi_writer:add(Out2=Out, Key2, Value2)
end,
scan(BT1, BT2, Out2, IsLastLevel, AT, BT, Count+1, step(Step))
end.
scan_only(BT, Out, IsLastLevel, [], Count, Step) ->
case lsm_btree_reader:next_node(BT) of
case hanoi_reader:next_node(BT) of
{node, KVs} ->
scan_only(BT, Out, IsLastLevel, KVs, Count, step(Step));
end_of_data ->
@ -150,8 +150,8 @@ scan_only(BT, Out, true, [{_,?TOMBSTONE}|Rest], Count, Step) ->
scan_only(BT, Out, IsLastLevel, [{Key,Value}|Rest], Count, Step) ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key, Value}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key, Value}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key, Value)
ok = hanoi_writer:add(Out2=Out, Key, Value)
end,
scan_only(BT, Out2, IsLastLevel, Rest, Count+1, step(Step)).

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,14 +22,14 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_nursery).
-module(hanoi_nursery).
-author('Kresten Krab Thorup <krab@trifork.com>').
-export([new/1, recover/2, add/3, finish/2, lookup/2, add_maybe_flush/4]).
-export([do_level_fold/3]).
-include("include/lsm_btree.hrl").
-include("lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("hanoi.hrl").
-include_lib("kernel/include/file.hrl").
-record(nursery, { log_file, dir, cache, total_size=0, count=0 }).
@ -147,14 +147,14 @@ finish(#nursery{ dir=Dir, cache=Cache, log_file=LogFile, total_size=_TotalSize,
N when N>0 ->
%% next, flush cache to a new BTree
BTreeFileName = filename:join(Dir, "nursery.data"),
{ok, BT} = lsm_btree_writer:open(BTreeFileName, ?BTREE_SIZE(?TOP_LEVEL)),
{ok, BT} = hanoi_writer:open(BTreeFileName, ?BTREE_SIZE(?TOP_LEVEL)),
try
lists:foreach( fun({Key,Value}) ->
ok = lsm_btree_writer:add(BT, Key, Value)
ok = hanoi_writer:add(BT, Key, Value)
end,
gb_trees:to_list(Cache))
after
ok = lsm_btree_writer:close(BT)
ok = hanoi_writer:close(BT)
end,
% {ok, FileInfo} = file:read_file_info(BTreeFileName),
@ -162,11 +162,11 @@ finish(#nursery{ dir=Dir, cache=Cache, log_file=LogFile, total_size=_TotalSize,
% [ gb_trees:size(Cache), TotalSize, FileInfo#file_info.size ]),
%% inject the B-Tree (blocking RPC)
ok = lsm_btree_level:inject(TopLevel, BTreeFileName),
ok = hanoi_level:inject(TopLevel, BTreeFileName),
%% issue some work if this is a top-level inject (blocks until previous such
%% incremental merge is finished).
lsm_btree_level:incremental_merge(TopLevel, ?BTREE_SIZE(?TOP_LEVEL)),
hanoi_level:incremental_merge(TopLevel, ?BTREE_SIZE(?TOP_LEVEL)),
ok;
_ ->
@ -183,9 +183,9 @@ add_maybe_flush(Key, Value, Nursery=#nursery{ dir=Dir }, Top) ->
{ok, _} = OK ->
OK;
{full, Nursery2} ->
ok = lsm_btree_nursery:finish(Nursery2, Top),
ok = hanoi_nursery:finish(Nursery2, Top),
{error, enoent} = file:read_file_info( filename:join(Dir, "nursery.log")),
lsm_btree_nursery:new(Dir)
hanoi_nursery:new(Dir)
end.
do_level_fold(#nursery{ cache=Cache }, FoldWorkerPID, KeyRange) ->

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,12 +22,12 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_reader).
-module(hanoi_reader).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include_lib("kernel/include/file.hrl").
-include("include/lsm_btree.hrl").
-include("lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("hanoi.hrl").
-export([open/1, open/2,close/1,lookup/2,fold/3,range_fold/4]).
-export([first_node/1,next_node/1]).
@ -232,7 +232,7 @@ find(_, _) ->
read_node(File,{Pos,Size}) ->
% error_logger:info_msg("read_node ~p ~p ~p~n", [File, Pos, Size]),
{ok, <<_:32, Level:16/unsigned, Data/binary>>} = file:pread(File, Pos, Size),
lsm_btree_util:decode_index_node(Level, Data);
hanoi_util:decode_index_node(Level, Data);
read_node(File,Pos) ->
{ok, Pos} = file:position(File, Pos),
@ -246,7 +246,7 @@ read_node(File) ->
0 -> eof;
_ ->
{ok, Data} = file:read(File, Len-2),
{ok, Node} = lsm_btree_util:decode_index_node(Level, Data),
{ok, Node} = hanoi_util:decode_index_node(Level, Data),
{ok, Node}
end.
@ -257,7 +257,7 @@ next_leaf_node(File) ->
eof;
{ok, <<Len:32, 0:16>>} ->
{ok, Data} = file:read(File, Len-2),
lsm_btree_util:decode_index_node(0, Data);
hanoi_util:decode_index_node(0, Data);
{ok, <<Len:32, _:16>>} ->
{ok, _} = file:position(File, {cur,Len-2}),
next_leaf_node(File)

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_sup).
-module(hanoi_sup).
-author('Kresten Krab Thorup <krab@trifork.com>').
-behaviour(supervisor).

View file

@ -20,14 +20,14 @@
%%% NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE
%%%
%%% This is a temporary copy of riak_kv_backend, just here to keep lsm_btree
%%% development private for now. When riak_kv_lsm_btree_backend is moved to
%%% This is a temporary copy of riak_kv_backend, just here to keep hanoi
%%% development private for now. When riak_kv_hanoi_backend is moved to
%%% riak_kv, delete this file.
%%%
%%% NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE
-module(lsm_btree_temp_riak_kv_backend).
-module(hanoi_temp_riak_kv_backend).
-export([behaviour_info/1]).
-export([callback_after/3]).

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_util).
-module(hanoi_util).
-author('Kresten Krab Thorup <krab@trifork.com>').
-compile(export_all).

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,10 +22,10 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_writer).
-module(hanoi_writer).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include("lsm_btree.hrl").
-include("hanoi.hrl").
%%
%% Streaming btree writer. Accepts only monotonically increasing keys for put.
@ -80,7 +80,7 @@ init([Name,Size]) ->
% io:format("got name: ~p~n", [Name]),
case file:open( lsm_btree_util:index_file_name(Name),
case file:open( hanoi_util:index_file_name(Name),
[raw, exclusive, write, {delayed_write, 512 * 1024, 2000}]) of
{ok, IdxFile} ->
{ok, BloomFilter} = ebloom:new(erlang:min(Size,16#ffffffff), 0.01, 123),
@ -89,7 +89,7 @@ init([Name,Size]) ->
bloom = BloomFilter
}};
{error, _}=Error ->
error_logger:error_msg("lsm_btree_writer cannot open ~p: ~p~n", [Name, Error]),
error_logger:error_msg("hanoi_writer cannot open ~p: ~p~n", [Name, Error]),
{stop, Error}
end.
@ -113,7 +113,7 @@ terminate(normal,_State) ->
%% premature delete -> cleanup
terminate(_Reason,State) ->
file:close( State#state.index_file ),
file:delete( lsm_btree_util:index_file_name(State#state.name) ).
file:delete( hanoi_util:index_file_name(State#state.name) ).
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
@ -162,7 +162,7 @@ add_record(Level, Key, Value,
end
end,
NewSize = NodeSize + lsm_btree_util:estimate_node_size_increment(List, Key, Value),
NewSize = NodeSize + hanoi_util:estimate_node_size_increment(List, Key, Value),
ebloom:insert( State#state.bloom, Key ),
@ -182,7 +182,7 @@ add_record(Level, Key, Value, #state{ nodes=Nodes }=State) ->
close_node(#state{nodes=[#node{ level=Level, members=NodeMembers }|RestNodes]} = State) ->
OrderedMembers = lists:reverse(NodeMembers),
{ok, DataSize, Data} = lsm_btree_util:encode_index_node(Level, OrderedMembers),
{ok, DataSize, Data} = hanoi_util:encode_index_node(Level, OrderedMembers),
NodePos = State#state.index_file_pos,
ok = file:write(State#state.index_file, Data),

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2012 (c) Basho Technologies, Inc. All Rights Reserved.
%% http://basho.com/ info@basho.com
@ -19,8 +19,8 @@
%%
%% ----------------------------------------------------------------------------
-module(riak_kv_lsm_btree_backend).
-behavior(lsm_btree_temp_riak_kv_backend).
-module(riak_kv_hanoi_backend).
-behavior(hanoi_temp_riak_kv_backend).
-author('Steve Vinoski <steve@basho.com>').
-author('Greg Burd <greg@basho.com>').
@ -45,7 +45,7 @@
-include_lib("eunit/include/eunit.hrl").
-endif.
-include("include/lsm_btree.hrl").
-include("include/hanoi.hrl").
-define(API_VERSION, 1).
%% TODO: for when this backend supports 2i
@ -78,37 +78,37 @@ capabilities(_) ->
capabilities(_, _) ->
{ok, ?CAPABILITIES}.
%% @doc Start the lsm_btree backend
%% @doc Start the hanoi backend
-spec start(integer(), config()) -> {ok, state()} | {error, term()}.
start(Partition, Config) ->
%% Get the data root directory
case app_helper:get_prop_or_env(data_root, Config, lsm_btree) of
case app_helper:get_prop_or_env(data_root, Config, hanoi) of
undefined ->
lager:error("Failed to create lsm_btree dir: data_root is not set"),
lager:error("Failed to create hanoi dir: data_root is not set"),
{error, data_root_unset};
DataRoot ->
AppStart = case application:start(lsm_btree) of
AppStart = case application:start(hanoi) of
ok ->
ok;
{error, {already_started, _}} ->
ok;
{error, StartReason} ->
lager:error("Failed to init the lsm_btree backend: ~p", [StartReason]),
lager:error("Failed to init the hanoi backend: ~p", [StartReason]),
{error, StartReason}
end,
case AppStart of
ok ->
case get_data_dir(DataRoot, integer_to_list(Partition)) of
{ok, DataDir} ->
case lsm_btree:open(DataDir) of
case hanoi:open(DataDir) of
{ok, Tree} ->
{ok, #state{tree=Tree, partition=Partition}};
{error, OpenReason}=OpenError ->
lager:error("Failed to open lsm_btree: ~p\n", [OpenReason]),
lager:error("Failed to open hanoi: ~p\n", [OpenReason]),
OpenError
end;
{error, Reason} ->
lager:error("Failed to start lsm_btree backend: ~p\n", [Reason]),
lager:error("Failed to start hanoi backend: ~p\n", [Reason]),
{error, Reason}
end;
Error ->
@ -116,19 +116,19 @@ start(Partition, Config) ->
end
end.
%% @doc Stop the lsm_btree backend
%% @doc Stop the hanoi backend
-spec stop(state()) -> ok.
stop(#state{tree=Tree}) ->
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
%% @doc Retrieve an object from the lsm_btree backend
%% @doc Retrieve an object from the hanoi backend
-spec get(riak_object:bucket(), riak_object:key(), state()) ->
{ok, any(), state()} |
{ok, not_found, state()} |
{error, term(), state()}.
get(Bucket, Key, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
case lsm_btree:get(Tree, BKey) of
case hanoi:get(Tree, BKey) of
{ok, Value} ->
{ok, Value, State};
not_found ->
@ -137,23 +137,23 @@ get(Bucket, Key, #state{tree=Tree}=State) ->
{error, Reason, State}
end.
%% @doc Insert an object into the lsm_btree backend.
%% @doc Insert an object into the hanoi backend.
-type index_spec() :: {add, Index, SecondaryKey} | {remove, Index, SecondaryKey}.
-spec put(riak_object:bucket(), riak_object:key(), [index_spec()], binary(), state()) ->
{ok, state()} |
{error, term(), state()}.
put(Bucket, Key, _IndexSpecs, Val, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
ok = lsm_btree:put(Tree, BKey, Val),
ok = hanoi:put(Tree, BKey, Val),
{ok, State}.
%% @doc Delete an object from the lsm_btree backend
%% @doc Delete an object from the hanoi backend
-spec delete(riak_object:bucket(), riak_object:key(), [index_spec()], state()) ->
{ok, state()} |
{error, term(), state()}.
delete(Bucket, Key, _IndexSpecs, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
case lsm_btree:delete(Tree, BKey) of
case hanoi:delete(Tree, BKey) of
ok ->
{ok, State};
{error, Reason} ->
@ -170,7 +170,7 @@ fold_buckets(FoldBucketsFun, Acc, Opts, #state{tree=Tree}) ->
BucketFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, {Acc, []}, #btree_range{})
hanoi:sync_fold_range(Tree, FoldFun, {Acc, []}, #btree_range{})
catch
{break, AccFinal} ->
AccFinal
@ -213,7 +213,7 @@ fold_keys(FoldKeysFun, Acc, Opts, #state{tree=Tree}) ->
KeyFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, Acc, Range)
hanoi:sync_fold_range(Tree, FoldFun, Acc, Range)
catch
{break, AccFinal} ->
AccFinal
@ -250,7 +250,7 @@ fold_objects(FoldObjectsFun, Acc, Opts, #state{tree=Tree}) ->
ObjectFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, Acc, bucket_range(Bucket))
hanoi:sync_fold_range(Tree, FoldFun, Acc, bucket_range(Bucket))
catch
{break, AccFinal} ->
AccFinal
@ -263,26 +263,26 @@ fold_objects(FoldObjectsFun, Acc, Opts, #state{tree=Tree}) ->
{ok, ObjectFolder()}
end.
%% @doc Delete all objects from this lsm_btree backend
%% @doc Delete all objects from this hanoi backend
-spec drop(state()) -> {ok, state()} | {error, term(), state()}.
drop(#state{}=State) ->
%% TODO: not yet implemented
{ok, State}.
%% @doc Returns true if this lsm_btree backend contains any
%% @doc Returns true if this hanoi backend contains any
%% non-tombstone values; otherwise returns false.
-spec is_empty(state()) -> boolean().
is_empty(#state{tree=Tree}) ->
FoldFun = fun(_K, _V, _Acc) -> throw(ok) end,
try
Range = #btree_range{},
[] =:= lsm_btree:sync_fold_range(Tree, FoldFun, [], Range)
[] =:= hanoi:sync_fold_range(Tree, FoldFun, [], Range)
catch
_:ok ->
false
end.
%% @doc Get the status information for this lsm_btree backend
%% @doc Get the status information for this hanoi backend
-spec status(state()) -> [{atom(), term()}].
status(#state{}) ->
%% TODO: not yet implemented
@ -306,7 +306,7 @@ get_data_dir(DataRoot, Partition) ->
ok ->
{ok, PartitionDir};
{error, Reason} ->
lager:error("Failed to create lsm_btree dir ~s: ~p", [PartitionDir, Reason]),
lager:error("Failed to create hanoi dir ~s: ~p", [PartitionDir, Reason]),
{error, Reason}
end.
@ -377,13 +377,13 @@ from_object_key(LKey) ->
-ifdef(TEST).
simple_test_() ->
?assertCmd("rm -rf test/lsm-btree-backend"),
application:set_env(lsm_btree, data_root, "test/lsm-btree-backend"),
lsm_btree_temp_riak_kv_backend:standard_test(?MODULE, []).
?assertCmd("rm -rf test/hanoi-backend"),
application:set_env(hanoi, data_root, "test/hanoid-backend"),
hanoi_temp_riak_kv_backend:standard_test(?MODULE, []).
custom_config_test_() ->
?assertCmd("rm -rf test/lsm_btree-backend"),
application:set_env(lsm_btree, data_root, ""),
lsm_btree_temp_riak_kv_backend:standard_test(?MODULE, [{data_root, "test/lsm-btree-backend"}]).
?assertCmd("rm -rf test/hanoi-backend"),
application:set_env(hanoi, data_root, ""),
hanoi_temp_riak_kv_backend:standard_test(?MODULE, [{data_root, "test/hanoi-backend"}]).
-endif.

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,8 +22,8 @@
%%
%% ----------------------------------------------------------------------------
%% @Doc Drive a set of lsm BTrees
-module(lsm_btree_drv).
%% @Doc Drive a set of LSM BTrees
-module(hanoi_drv).
-behaviour(gen_server).
@ -90,7 +90,7 @@ init([]) ->
{ok, #state{}}.
handle_call({open, N}, _, #state { btrees = D} = State) ->
case lsm_btree:open(N) of
case hanoi:open(N) of
{ok, Tree} ->
{reply, ok, State#state { btrees = dict:store(N, Tree, D)}};
Otherwise ->
@ -98,7 +98,7 @@ handle_call({open, N}, _, #state { btrees = D} = State) ->
end;
handle_call({close, N}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
case lsm_btree:close(Tree) of
case hanoi:close(Tree) of
ok ->
{reply, ok, State#state { btrees = dict:erase(N, D)}};
Otherwise ->
@ -107,18 +107,18 @@ handle_call({close, N}, _, #state { btrees = D} = State) ->
handle_call({sync_range, Name, Range}, _From,
#state { btrees = D} = State) ->
Tree = dict:fetch(Name, D),
{ok, Ref} = lsm_btree:sync_range(Tree, Range),
{ok, Ref} = hanoi:sync_range(Tree, Range),
Result = sync_range_gather(Ref),
{reply, Result, State};
handle_call({sync_fold_range, Name, Fun, Acc0, Range},
_From,
#state { btrees = D } = State) ->
Tree = dict:fetch(Name, D),
Result = lsm_btree:sync_fold_range(Tree, Fun, Acc0, Range),
Result = hanoi:sync_fold_range(Tree, Fun, Acc0, Range),
{reply, Result, State};
handle_call({put, N, K, V}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
case lsm_btree:put(Tree, K, V) of
case hanoi:put(Tree, K, V) of
ok ->
{reply, ok, State};
Other ->
@ -126,14 +126,14 @@ handle_call({put, N, K, V}, _, #state { btrees = D} = State) ->
end;
handle_call({delete_exist, N, K}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
Reply = lsm_btree:delete(Tree, K),
Reply = hanoi:delete(Tree, K),
{reply, Reply, State};
handle_call({get, N, K}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
Reply = lsm_btree:get(Tree, K),
Reply = hanoi:get(Tree, K),
{reply, Reply, State};
handle_call(stop, _, #state{ btrees = D } = State ) ->
[ lsm_btree:close(Tree) || {_,Tree} <- dict:to_list(D) ],
[ hanoi:close(Tree) || {_,Tree} <- dict:to_list(D) ],
{stop, normal, ok, State};
handle_call(_Request, _From, State) ->
Reply = ok,

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_merger_tests).
-module(hanoi_merger_tests).
-ifdef(TEST).
-include_lib("proper/include/proper.hrl").
@ -37,26 +37,26 @@ merge_test() ->
file:delete("test2"),
file:delete("test3"),
{ok, BT1} = lsm_btree_writer:open("test1"),
{ok, BT1} = hanoi_writer:open("test1"),
lists:foldl(fun(N,_) ->
ok = lsm_btree_writer:add(BT1, <<N:128>>, <<"data",N:128>>)
ok = hanoi_writer:add(BT1, <<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(1,10000,2)),
ok = lsm_btree_writer:close(BT1),
ok = hanoi_writer:close(BT1),
{ok, BT2} = lsm_btree_writer:open("test2"),
{ok, BT2} = hanoi_writer:open("test2"),
lists:foldl(fun(N,_) ->
ok = lsm_btree_writer:add(BT2, <<N:128>>, <<"data",N:128>>)
ok = hanoi_writer:add(BT2, <<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(2,5001,1)),
ok = lsm_btree_writer:close(BT2),
ok = hanoi_writer:close(BT2),
self() ! {step, {self(), none}, 2000000000},
{Time,{ok,Count}} = timer:tc(lsm_btree_merger, merge, ["test1", "test2", "test3", 10000, true]),
{Time,{ok,Count}} = timer:tc(hanoi_merger, merge, ["test1", "test2", "test3", 10000, true]),
error_logger:info_msg("time to merge: ~p/sec (time=~p, count=~p)~n", [1000000/(Time/Count), Time/1000000, Count]),

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,10 +22,10 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_tests).
-module(hanoi_tests).
-include("include/lsm_btree.hrl").
-include("src/lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("src/hanoi.hrl").
-ifdef(TEST).
-ifdef(TRIQ).
@ -50,7 +50,7 @@
-record(tree, { elements = dict:new() }).
-record(state, { open = dict:new(),
closed = dict:new() }).
-define(SERVER, lsm_btree_drv).
-define(SERVER, hanoi_drv).
full_test_() ->
{setup,
@ -272,9 +272,9 @@ prop_dict_agree() ->
?FORALL(Cmds, commands(?MODULE),
?TRAPEXIT(
begin
lsm_btree_drv:start_link(),
hanoi_drv:start_link(),
{History,State,Result} = run_commands(?MODULE, Cmds),
lsm_btree_drv:stop(),
hanoi_drv:stop(),
cleanup_test_trees(State),
?WHENFAIL(io:format("History: ~w\nState: ~w\nResult: ~w\n",
[History,State,Result]),
@ -284,64 +284,64 @@ prop_dict_agree() ->
%% UNIT TESTS
%% ----------------------------------------------------------------------
test_tree_simple_1() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<>>, <<"data", 77:128>>),
{ok, <<"data", 77:128>>} = lsm_btree:get(Tree, <<>>),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<>>, <<"data", 77:128>>),
{ok, <<"data", 77:128>>} = hanoi:get(Tree, <<>>),
ok = hanoi:close(Tree).
test_tree_simple_2() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<"ã">>, <<"µ">>),
ok = lsm_btree:delete(Tree, <<"ã">>),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<"ã">>, <<"µ">>),
ok = hanoi:delete(Tree, <<"ã">>),
ok = hanoi:close(Tree).
test_tree_simple_3() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<"X">>, <<"Y">>),
{ok, Ref} = lsm_btree:sync_range(Tree, #btree_range{from_key= <<"X">>, to_key= <<"X">>}),
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<"X">>, <<"Y">>),
{ok, Ref} = hanoi:sync_range(Tree, #btree_range{from_key= <<"X">>, to_key= <<"X">>}),
?assertEqual(ok,
receive
{fold_done, Ref} -> ok
after 1000 -> {error, timeout}
end),
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
test_tree_simple_4() ->
Key = <<56,11,62,42,35,163,16,100,9,224,8,228,130,94,198,2,126,117,243,
1,122,175,79,159,212,177,30,153,71,91,85,233,41,199,190,58,3,
173,220,9>>,
Value = <<212,167,12,6,105,152,17,80,243>>,
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, Key, Value),
?assertEqual({ok, Value}, lsm_btree:get(Tree, Key)),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, Key, Value),
?assertEqual({ok, Value}, hanoi:get(Tree, Key)),
ok = hanoi:close(Tree).
test_tree() ->
{ok, Tree} = lsm_btree:open("simple2"),
{ok, Tree} = hanoi:open("simple2"),
lists:foldl(fun(N,_) ->
ok = lsm_btree:put(Tree,
ok = hanoi:put(Tree,
<<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(2,100000,1)),
lists:foldl(fun(N,_) ->
ok = lsm_btree:put(Tree,
ok = hanoi:put(Tree,
<<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(4000,6000,1)),
lsm_btree:delete(Tree, <<1500:128>>),
hanoi:delete(Tree, <<1500:128>>),
{Time,{ok,Count}} = timer:tc(?MODULE, run_fold, [Tree,1000,2000]),
error_logger:info_msg("time to fold: ~p/sec (time=~p, count=~p)~n", [1000000/(Time/Count), Time/1000000, Count]),
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
run_fold(Tree,From,To) ->
{ok, PID} = lsm_btree:sync_range(Tree, #btree_range{from_key= <<From:128>>, to_key= <<(To+1):128>>}),
{ok, PID} = hanoi:sync_range(Tree, #btree_range{from_key= <<From:128>>, to_key= <<(To+1):128>>}),
lists:foreach(fun(1500) -> ok;
(N) ->
receive

View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_writer_tests).
-module(hanoi_writer_tests).
-ifdef(TEST).
-include_lib("proper/include/proper.hrl").
@ -33,21 +33,21 @@
simple_test() ->
{ok, BT} = lsm_btree_writer:open("testdata"),
ok = lsm_btree_writer:add(BT, <<"A">>, <<"Avalue">>),
ok = lsm_btree_writer:add(BT, <<"B">>, <<"Bvalue">>),
ok = lsm_btree_writer:close(BT),
{ok, BT} = hanoi_writer:open("testdata"),
ok = hanoi_writer:add(BT, <<"A">>, <<"Avalue">>),
ok = hanoi_writer:add(BT, <<"B">>, <<"Bvalue">>),
ok = hanoi_writer:close(BT),
{ok, IN} = lsm_btree_reader:open("testdata"),
{ok, <<"Avalue">>} = lsm_btree_reader:lookup(IN, <<"A">>),
ok = lsm_btree_reader:close(IN),
{ok, IN} = hanoi_reader:open("testdata"),
{ok, <<"Avalue">>} = hanoi_reader:lookup(IN, <<"A">>),
ok = hanoi_reader:close(IN),
ok = file:delete("testdata").
simple1_test() ->
{ok, BT} = lsm_btree_writer:open("testdata"),
{ok, BT} = hanoi_writer:open("testdata"),
Max = 30*1024,
Seq = lists:seq(0, Max),
@ -56,21 +56,21 @@ simple1_test() ->
fun() ->
lists:foreach(
fun(Int) ->
ok = lsm_btree_writer:add(BT, <<Int:128>>, <<"valuevalue/", Int:128>>)
ok = hanoi_writer:add(BT, <<Int:128>>, <<"valuevalue/", Int:128>>)
end,
Seq),
ok = lsm_btree_writer:close(BT)
ok = hanoi_writer:close(BT)
end,
[]),
error_logger:info_msg("time to insert: ~p/sec~n", [1000000/(Time1/Max)]),
{ok, IN} = lsm_btree_reader:open("testdata"),
{ok, <<"valuevalue/", 2048:128>>} = lsm_btree_reader:lookup(IN, <<2048:128>>),
{ok, IN} = hanoi_reader:open("testdata"),
{ok, <<"valuevalue/", 2048:128>>} = hanoi_reader:lookup(IN, <<2048:128>>),
{Time2,Count} = timer:tc(
fun() -> lsm_btree_reader:fold(fun(Key, <<"valuevalue/", Key/binary>>, N) ->
fun() -> hanoi_reader:fold(fun(Key, <<"valuevalue/", Key/binary>>, N) ->
N+1
end,
0,
@ -83,6 +83,6 @@ simple1_test() ->
Max = Count-1,
ok = lsm_btree_reader:close(IN),
ok = hanoi_reader:close(IN),
ok = file:delete("testdata").

View file

@ -2,7 +2,7 @@
## ----------------------------------------------------------------------------
##
## lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
## hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
##
## Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
## http://trifork.com/ info@trifork.com