greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity

Merge pull request #3 from basho/rename-to-hanoi

Browse source 
Rename "lsm_btree" to "hanoi".
This commit is contained in:
Gregory Burd 2012-04-21 12:30:07 -07:00
commit 638f2d56ee
27 changed files with 267 additions and 267 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
Makefile
View file

@ -28,7 +28,7 @@ clean-test-btrees:
rm -fr .eunit/Btree_* .eunit/simple
plt: compile
$(DIALYZER) --build_plt --output_plt .lsm_btree.plt \
$(DIALYZER) --build_plt --output_plt .hanoi.plt \
-pa deps/plain_fsm/ebin \
-pa deps/ebloom/ebin \
deps/plain_fsm/ebin \
@ -36,7 +36,7 @@ plt: compile
--apps kernel stdlib
analyze: compile
$(DIALYZER) --plt .lsm_btree.plt \
$(DIALYZER) --plt .hanoi.plt \
-pa deps/plain_fsm/ebin \
-pa deps/ebloom/ebin \
ebin

26
README.md
View file

@ -1,8 +1,8 @@
# LSM B-Tree Storage
# Hanoi Key/Value Storage Engine
This erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees and Fractal Trees indicates that this storage engine is quite different in several respects.
This Erlang-based storage engine implements a structure somewhat like LSM-trees (Log-Structured Merge Trees, see docs/10.1.1.44.2782.pdf). The notes below describe how this storage engine work; I have not done extensive studies as how it differs from other storage mechanisms, but a brief brows through available online resources on LSM-trees indicates that this storage engine is quite different in several respects.
The storage engine may eventually provide an alternative backend for Basho's Riak. Of the existing backends, `lsm_btree` is closest to LevelDB in operational characteristics, but it is implemented in just ~1000 lines of Erlang.
The storage engine can function as an alternative backend for Basho's Riak/KV.
Here's the bullet list:
@ -16,7 +16,7 @@ Here's the bullet list:
Once we're a bit more stable, we'll provide a Riak backend.
## How a LSM-BTree Works
## How this LSM-BTree Works
If there are N records, there are in log<sub>2</sub>(N) levels (each being a plain B-tree in a file named "A-*level*.data"). The file `A-0.data` has 1 record, `A-1.data` has 2 records, `A-2.data` has 4 records, and so on: `A-n.data` has 2<sup>n</sup> records.
@ -33,8 +33,8 @@ Insertion works by a mechanism known as B-tree injection. Insertion always star
- If the level being injected into empty (there is no A-*level*.data file), then the injected B-tree becomes the contents for that level (we just rename the file).
- Otherwise,
- The injected tree file is renamed to B-*level*.data;
- The files A-*level*.data and B-*level*.data are merged into a new temporary B-tree (of roughly double size), X-*level*.data.
- The outcome of the merge is then injected into the next level.
- The files A-*level*.data and B-*level*.data are merged into a new temporary B-tree (of roughly double size), X-*level*.data.
- The outcome of the merge is then injected into the next level.
While merging, lookups at level *n* first consults the B-*n*.data file, then the A-*n*.data file. At a given level, there can only be one merge operation active.
@ -55,16 +55,16 @@ A further trouble is that merging does in fact not have completely linear I/O co
Merging can be going on concurrently at each level (in preparation for an injection to the next level), which lets you utilize available multi-core capacity to merge.
### Deploying the lsm_btree for testing with Riak/KV
### Deploying the hanoi for testing with Riak/KV
You can deploy `lsm_btree` into a Riak devrel cluster using the
`enable-lsm_btree` script. Clone the `riak` repo, change your working directory
to it, and then execute the `enable-lsm_btree` script. It adds `lsm_btree` as a
You can deploy `hanoi` into a Riak devrel cluster using the
`enable-hanoi` script. Clone the `riak` repo, change your working directory
to it, and then execute the `enable-hanoi` script. It adds `hanoi` as a
dependency, runs `make all devrel`, and then modifies the configuration
settings of the resulting dev nodes to use the lsm_btree storage backend.
settings of the resulting dev nodes to use the hanoi storage backend.
1. `git clone git://github.com/basho/riak.git`
1. `cd riak/deps`
1. `git clone git://github.com/basho/lsm_btree.git`
1. `git clone git://github.com/basho/hanoi.git`
1. `cd ..`
1. `./deps/lsm_btree/enable-lsm_btree` # which does `make all devrel`
1. `./deps/hanoi/enable-hanoi` # which does `make all devrel`

12
TODO
View file

@ -1,4 +1,4 @@
* lsm_btree
* hanoi
* [cleanup] add @doc strings and and -spec's
* [cleanup] check to make sure every error returns with a reason {error, Reason}
* [feature] statistics
@ -23,7 +23,7 @@
serviced, then picks up from there again. So you could see intermittent
puts in a subsequent batch of results.
* riak_kv_lsm_tree_backend
* riak_kv_hanoie_backend
* add support for time-based expiry
* finish support for 2i
* add stats collection
@ -31,16 +31,16 @@
PHASE 2:
* lsm_btree
* hanoi
* Define a standard struct which is the metadata added at the end of the
file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written
in lsm_btree_writer:flush_nodes, and read in lsm_btree_reader:open2.
in hanoi_writer:flush_nodes, and read in hanoi_reader:open2.
* [feature] compression, encryption on disk
PHASE 3:
* lsm_ixdb
* lsm_{btree, ctrie, ...} support for sub-databases and associations with
* hanoi{btree, trie, ...} support for sub-databases and associations with
different index types
* [major change] add more CAPABILITIES such as
test-and-set(Fun, Key, Value) - to compare a vclock quickly, to speed up
@ -56,7 +56,7 @@ REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS:
* http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.44.2782&rep=rep1&type=pdf
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in lsm_btree.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in hanoi.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads.

0
doc/compare-innodb-vs-lsm_btree.png → doc/compare-innodb-vs-hanoi.png
View file

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 119 KiB

After

Width:  |  Height:  |  Size: 119 KiB

18
enable-lsm_btree → enable-hanoi
View file

@ -1,12 +1,12 @@
#!/bin/sh
# This script adds lsm_btree to a riak github repo. Run it in the riak repo
# This script adds hanoi to a riak github repo. Run it in the riak repo
# directory.
#
# First it adds lsm_btree, then runs "make all devrel" and then enables the
# lsm_btree storage backend in the resulting dev nodes.
# First it adds hanoi, then runs "make all devrel" and then enables the
# hanoi storage backend in the resulting dev nodes.
#
# This script is intended to be temporary. Once lsm_btree is made into a proper
# This script is intended to be temporary. Once hanoi is made into a proper
# riak citizen, this script will no longer be needed.
set -e
@ -35,17 +35,17 @@ fi
./rebar get-deps
file=./deps/riak_kv/src/riak_kv.app.src
if ! grep -q lsm_btree $file ; then
if ! grep -q hanoi $file ; then
echo
echo "Modifying $file, saving the original as ${file}.orig ..."
perl -i.orig -pe '/\bos_mon,/ && print qq( lsm_btree,\n)' $file
perl -i.orig -pe '/\bos_mon,/ && print qq( hanoi,\n)' $file
fi
file=./deps/riak_kv/rebar.config
if ! grep -q lsm_btree $file ; then
if ! grep -q hanoi $file ; then
echo
echo "Modifying $file, saving the original as ${file}.orig ..."
perl -i.orig -pe '/\bsext\b/ && print qq( {lsm_btree, ".*", {git, "git\@github.com:basho/lsm_btree.git", "master"}},\n)' $file
perl -i.orig -pe '/\bsext\b/ && print qq( {hanoi, ".*", {git, "git\@github.com:basho/hanoi.git", "master"}},\n)' $file
fi
./rebar get-deps
@ -55,6 +55,6 @@ make all devrel
echo
echo 'Modifying all dev/dev*/etc/app.config files, saving originals with .orig suffix...'
perl -i.orig -ne 'if (/\bstorage_backend,/) { s/(storage_backend, )[^\}]+/\1riak_kv_lsm_btree_backend/; print } elsif (/\{eleveldb,/) { $eleveldb++; print } elsif ($eleveldb && /^\s+\]\},/) { $eleveldb = 0; print; print qq(\n {lsm_btree, [\n {data_root, "./data/lsm_btree"}\n ]},\n\n) } else { print }' dev/dev*/etc/app.config
perl -i.orig -ne 'if (/\bstorage_backend,/) { s/(storage_backend, )[^\}]+/\1riak_kv_hanoi_backend/; print } elsif (/\{eleveldb,/) { $eleveldb++; print } elsif ($eleveldb && /^\s+\]\},/) { $eleveldb = 0; print; print qq(\n {hanoi, [\n {data_root, "./data/hanoi"}\n ]},\n\n) } else { print }' dev/dev*/etc/app.config
exit 0

2
include/lsm_btree.hrl → include/hanoi.hrl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com

28
src/basho_bench_driver_lsm_btree.erl → src/basho_bench_driver_hanoi.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(basho_bench_driver_lsm_btree).
-module(basho_bench_driver_hanoi).
-record(state, { tree,
filename,
@ -33,7 +33,7 @@
-export([new/1,
run/4]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-include_lib("basho_bench/include/basho_bench.hrl").
-record(btree_range, { from_key = <<>> :: binary(),
@ -48,20 +48,20 @@
new(_Id) ->
%% Make sure bitcask is available
case code:which(lsm_btree) of
case code:which(hanoi) of
non_existing ->
?FAIL_MSG("~s requires lsm_btree to be available on code path.\n",
?FAIL_MSG("~s requires hanoi to be available on code path.\n",
[?MODULE]);
_ ->
ok
end,
%% Get the target directory
Dir = basho_bench_config:get(lsm_btree_dir, "."),
Filename = filename:join(Dir, "test.lsm_btree"),
Dir = basho_bench_config:get(hanoi_dir, "."),
Filename = filename:join(Dir, "test.hanoi"),
%% Look for sync interval config
case basho_bench_config:get(lsm_btree_sync_interval, infinity) of
case basho_bench_config:get(hanoi_sync_interval, infinity) of
Value when is_integer(Value) ->
SyncInterval = Value;
infinity ->
@ -69,9 +69,9 @@ new(_Id) ->
end,
%% Get any bitcask flags
case lsm_btree:open(Filename) of
case hanoi:open(Filename) of
{error, Reason} ->
?FAIL_MSG("Failed to open lsm btree in ~s: ~p\n", [Filename, Reason]);
?FAIL_MSG("Failed to open hanoi in ~s: ~p\n", [Filename, Reason]);
{ok, FBTree} ->
{ok, #state { tree = FBTree,
filename = Filename,
@ -80,7 +80,7 @@ new(_Id) ->
end.
run(get, KeyGen, _ValueGen, State) ->
case lsm_btree:lookup(State#state.tree, KeyGen()) of
case hanoi:lookup(State#state.tree, KeyGen()) of
{ok, _Value} ->
{ok, State};
not_found ->
@ -89,14 +89,14 @@ run(get, KeyGen, _ValueGen, State) ->
{error, Reason}
end;
run(put, KeyGen, ValueGen, State) ->
case lsm_btree:put(State#state.tree, KeyGen(), ValueGen()) of
case hanoi:put(State#state.tree, KeyGen(), ValueGen()) of
ok ->
{ok, State};
{error, Reason} ->
{error, Reason}
end;
run(delete, KeyGen, _ValueGen, State) ->
case lsm_btree:delete(State#state.tree, KeyGen()) of
case hanoi:delete(State#state.tree, KeyGen()) of
ok ->
{ok, State};
{error, Reason} ->
@ -105,7 +105,7 @@ run(delete, KeyGen, _ValueGen, State) ->
run(fold_100, KeyGen, _ValueGen, State) ->
[From,To] = lists:usort([KeyGen(), KeyGen()]),
case lsm_btree:sync_fold_range(State#state.tree,
case hanoi:sync_fold_range(State#state.tree,
fun(_Key,_Value,Count) ->
Count+1
end,

6
src/lsm_btree.app.src → src/hanoi.app.src
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
{application, lsm_btree,
{application, hanoi,
[
{description, ""},
{vsn, "1.0.0"},
@ -31,6 +31,6 @@
kernel,
stdlib
]},
{mod, {lsm_btree_app, []}},
{mod, {hanoi_app, []}},
{env, []}
]}.

44
src/lsm_btree.erl → src/hanoi.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree).
-module(hanoi).
-author('Kresten Krab Thorup <krab@trifork.com>').
@ -34,9 +34,9 @@
-export([open/1, close/1, get/2, lookup/2, delete/2, put/3,
async_range/2, async_fold_range/4, sync_range/2, sync_fold_range/4]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-include_lib("kernel/include/file.hrl").
-include_lib("include/lsm_btree.hrl").
-include_lib("include/hanoi.hrl").
-record(state, { top, nursery, dir }).
@ -87,7 +87,7 @@ sync_receive_fold_range(MRef, PID,Fun,Acc0) ->
{ok, Fun(K,V,Acc0)}
catch
Class:Exception ->
lager:warning("Exception in lsm_btree fold: ~p", [Exception]),
lager:warning("Exception in hanoi fold: ~p", [Exception]),
{'EXIT', Class, Exception, erlang:get_stacktrace()}
end
of
@ -106,7 +106,7 @@ sync_receive_fold_range(MRef, PID,Fun,Acc0) ->
{ok, kvfoldl(Fun,Acc0,KVs)}
catch
Class:Exception ->
lager:warning("Exception in lsm_btree fold: ~p", [Exception]),
lager:warning("Exception in hanoi fold: ~p", [Exception]),
{'EXIT', Class, Exception, erlang:get_stacktrace()}
end
of
@ -180,12 +180,12 @@ init([Dir]) ->
case file:read_file_info(Dir) of
{ok, #file_info{ type=directory }} ->
{ok, TopLevel} = open_levels(Dir),
{ok, Nursery} = lsm_btree_nursery:recover(Dir, TopLevel);
{ok, Nursery} = hanoi_nursery:recover(Dir, TopLevel);
{error, E} when E =:= enoent ->
ok = file:make_dir(Dir),
{ok, TopLevel} = lsm_btree_level:open(Dir, ?TOP_LEVEL, undefined),
{ok, Nursery} = lsm_btree_nursery:new(Dir)
{ok, TopLevel} = hanoi_level:open(Dir, ?TOP_LEVEL, undefined),
{ok, Nursery} = hanoi_nursery:new(Dir)
end,
{ok, #state{ top=TopLevel, dir=Dir, nursery=Nursery }}.
@ -216,7 +216,7 @@ open_levels(Dir) ->
TopLevel =
lists:foldl( fun(LevelNo, Prev) ->
{ok, Level} = lsm_btree_level:open(Dir,LevelNo,Prev),
{ok, Level} = hanoi_level:open(Dir,LevelNo,Prev),
Level
end,
undefined,
@ -256,15 +256,15 @@ code_change(_OldVsn, State, _Extra) ->
handle_call({async_range, Sender, Range}, _From, State=#state{ top=TopLevel, nursery=Nursery }) ->
{ok, FoldWorkerPID} = lsm_btree_fold_worker:start(Sender),
lsm_btree_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = lsm_btree_level:async_range(TopLevel, FoldWorkerPID, Range),
{ok, FoldWorkerPID} = hanoi_fold_worker:start(Sender),
hanoi_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = hanoi_level:async_range(TopLevel, FoldWorkerPID, Range),
{reply, Result, State};
handle_call({sync_range, Sender, Range}, _From, State=#state{ top=TopLevel, nursery=Nursery }) ->
{ok, FoldWorkerPID} = lsm_btree_fold_worker:start(Sender),
lsm_btree_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = lsm_btree_level:sync_range(TopLevel, FoldWorkerPID, Range),
{ok, FoldWorkerPID} = hanoi_fold_worker:start(Sender),
hanoi_nursery:do_level_fold(Nursery, FoldWorkerPID, Range),
Result = hanoi_level:sync_range(TopLevel, FoldWorkerPID, Range),
{reply, Result, State};
handle_call({put, Key, Value}, _From, State) when is_binary(Key), is_binary(Value) ->
@ -276,20 +276,20 @@ handle_call({delete, Key}, _From, State) when is_binary(Key) ->
{reply, ok, State2};
handle_call({get, Key}, _From, State=#state{ top=Top, nursery=Nursery } ) when is_binary(Key) ->
case lsm_btree_nursery:lookup(Key, Nursery) of
case hanoi_nursery:lookup(Key, Nursery) of
{value, ?TOMBSTONE} ->
{reply, not_found, State};
{value, Value} when is_binary(Value) ->
{reply, {ok, Value}, State};
none ->
Reply = lsm_btree_level:lookup(Top, Key),
Reply = hanoi_level:lookup(Top, Key),
{reply, Reply, State}
end;
handle_call(close, _From, State=#state{top=Top}) ->
try
{ok, State2} = flush_nursery(State),
ok = lsm_btree_level:close(Top),
ok = hanoi_level:close(Top),
{stop, normal, ok, State2}
catch
E:R ->
@ -298,10 +298,10 @@ handle_call(close, _From, State=#state{top=Top}) ->
end.
do_put(Key, Value, State=#state{ nursery=Nursery, top=Top }) ->
{ok, Nursery2} = lsm_btree_nursery:add_maybe_flush(Key, Value, Nursery, Top),
{ok, Nursery2} = hanoi_nursery:add_maybe_flush(Key, Value, Nursery, Top),
{ok, State#state{ nursery=Nursery2 }}.
flush_nursery(State=#state{nursery=Nursery, top=Top, dir=Dir}) ->
ok = lsm_btree_nursery:finish(Nursery, Top),
{ok, Nursery2} = lsm_btree_nursery:new(Dir),
ok = hanoi_nursery:finish(Nursery, Top),
{ok, Nursery2} = hanoi_nursery:new(Dir),
{ok, State#state{ nursery=Nursery2 }}.

2
src/lsm_btree.hrl → src/hanoi.hrl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com

6
src/lsm_btree_app.erl → src/hanoi_app.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_app).
-module(hanoi_app).
-author('Kresten Krab Thorup <krab@trifork.com>').
-behaviour(application).
@ -35,7 +35,7 @@
%% ===================================================================
start(_StartType, _StartArgs) ->
lsm_btree_sup:start_link().
hanoi_sup:start_link().
stop(_State) ->
ok.

12
src/lsm_btree_bbench.config → src/hanoi_bbench.config
View file

@ -5,7 +5,7 @@
{concurrent, 1}.
{driver, basho_bench_driver_lsm_btree}.
{driver, basho_bench_driver_hanoi}.
{key_generator, {int_to_bin,{uniform_int, 5000000}}}.
@ -14,13 +14,13 @@
{operations, [{get, 1}, {put, 1}]}.
%% the second element in the list below (e.g., "../../public/bitcask") must point to
%% the relevant directory of a lsm_btree installation
%% the relevant directory of a hanoi installation
{code_paths, ["deps/stats",
"../lsm_btree/ebin",
"../lsm_btree/deps/plain_fsm/ebin",
"../lsm_btree/deps/ebloom/ebin"
"../hanoi/ebin",
"../hanoi/deps/plain_fsm/ebin",
"../hanoi/deps/ebloom/ebin"
]}.
{bitcask_dir, "/tmp/lsm_btree.bench"}.
{bitcask_dir, "/tmp/hanoi.bench"}.
{bitcask_flags, [o_sync]}.

6
src/lsm_btree_fold_worker.erl → src/hanoi_fold_worker.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_fold_worker).
-module(hanoi_fold_worker).
-author('Kresten Krab Thorup <krab@trifork.com>').
@ -59,7 +59,7 @@
-behavior(plain_fsm).
-export([data_vsn/0, code_change/3]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
-record(state, {sendto}).

44
src/lsm_btree_level.erl → src/hanoi_level.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,14 +22,14 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_level).
-module(hanoi_level).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include("include/lsm_btree.hrl").
-include("src/lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("src/hanoi.hrl").
%%
%% Manages a "pair" of lsm_index (or rathern, 0, 1 or 2), and governs
%% Manages a "pair" of hanoi index (or rathern, 0, 1 or 2), and governs
%% the process of injecting/merging parent trees into this pair.
%%
@ -162,12 +162,12 @@ initialize2(State) ->
file:delete(BFileName),
ok = file:rename(MFileName, AFileName),
{ok, BT} = lsm_btree_reader:open(AFileName, random),
{ok, BT} = hanoi_reader:open(AFileName, random),
case file:read_file_info(CFileName) of
{ok, _} ->
file:rename(CFileName, BFileName),
{ok, BT2} = lsm_btree_reader:open(BFileName, random),
{ok, BT2} = hanoi_reader:open(BFileName, random),
check_begin_merge_then_loop(State#state{ a= BT, b=BT2 });
{error, enoent} ->
@ -177,12 +177,12 @@ initialize2(State) ->
{error, enoent} ->
case file:read_file_info(BFileName) of
{ok, _} ->
{ok, BT1} = lsm_btree_reader:open(AFileName, random),
{ok, BT2} = lsm_btree_reader:open(BFileName, random),
{ok, BT1} = hanoi_reader:open(AFileName, random),
{ok, BT2} = hanoi_reader:open(BFileName, random),
case file:read_file_info(CFileName) of
{ok, _} ->
{ok, BT3} = lsm_btree_reader:open(CFileName, random);
{ok, BT3} = hanoi_reader:open(CFileName, random);
{error, enoent} ->
BT3 = undefined
end,
@ -193,7 +193,7 @@ initialize2(State) ->
case file:read_file_info(AFileName) of
{ok, _} ->
{ok, BT1} = lsm_btree_reader:open(AFileName, random),
{ok, BT1} = hanoi_reader:open(AFileName, random),
main_loop(State#state{ a=BT1 });
{error, enoent} ->
@ -236,7 +236,7 @@ main_loop(State = #state{ next=Next }) ->
SetPos = #state.c
end,
ok = file:rename(FileName, ToFileName),
{ok, BT} = lsm_btree_reader:open(ToFileName, random),
{ok, BT} = hanoi_reader:open(ToFileName, random),
reply(From, ok),
check_begin_merge_then_loop(setelement(SetPos, State, BT));
@ -332,7 +332,7 @@ main_loop(State = #state{ next=Next }) ->
%% rpc would fail when we fall off the cliff
if Next == undefined -> ok;
true ->
lsm_btree_level:close(Next)
hanoi_level:close(Next)
end,
ok;
@ -445,7 +445,7 @@ main_loop(State = #state{ next=Next }) ->
% then, rename M to A, and open it
AFileName = filename("A",State2),
ok = file:rename(MFileName, AFileName),
{ok, BT} = lsm_btree_reader:open(AFileName, random),
{ok, BT} = hanoi_reader:open(AFileName, random),
% iff there is a C file, then move it to B position
% TODO: consider recovery for this
@ -535,7 +535,7 @@ do_lookup(_Key, [Pid]) when is_pid(Pid) ->
do_lookup(Key, [undefined|Rest]) ->
do_lookup(Key, Rest);
do_lookup(Key, [BT|Rest]) ->
case lsm_btree_reader:lookup(BT, Key) of
case hanoi_reader:lookup(BT, Key) of
{ok, ?TOMBSTONE} ->
not_found;
{ok, Result} ->
@ -545,7 +545,7 @@ do_lookup(Key, [BT|Rest]) ->
end.
close_if_defined(undefined) -> ok;
close_if_defined(BT) -> lsm_btree_reader:close(BT).
close_if_defined(BT) -> hanoi_reader:close(BT).
stop_if_defined(undefined) -> ok;
stop_if_defined(MergePid) when is_pid(MergePid) ->
@ -567,7 +567,7 @@ begin_merge(State) ->
file:delete(XFileName),
MergePID = proc_lib:spawn_link(fun() ->
{ok, OutCount} = lsm_btree_merger:merge(AFileName, BFileName, XFileName,
{ok, OutCount} = hanoi_merger:merge(AFileName, BFileName, XFileName,
?BTREE_SIZE(State#state.level + 1),
State#state.next =:= undefined),
% error_logger:info_msg("merge done ~p,~p -> ~p~n", [AFileName, BFileName, XFileName]),
@ -582,8 +582,8 @@ close_and_delete_a_and_b(State) ->
AFileName = filename("A",State),
BFileName = filename("B",State),
ok = lsm_btree_reader:close(State#state.a),
ok = lsm_btree_reader:close(State#state.b),
ok = hanoi_reader:close(State#state.a),
ok = hanoi_reader:close(State#state.b),
ok = file:delete(AFileName),
ok = file:delete(BFileName),
@ -600,7 +600,7 @@ start_range_fold(FileName, WorkerPID, Range) ->
PID =
proc_lib:spawn( fun() ->
erlang:link(WorkerPID),
{ok, File} = lsm_btree_reader:open(FileName, sequential),
{ok, File} = hanoi_reader:open(FileName, sequential),
do_range_fold(File, WorkerPID, self(), Range),
erlang:unlink(WorkerPID),
@ -609,12 +609,12 @@ start_range_fold(FileName, WorkerPID, Range) ->
end ),
{ok, PID}.
-spec do_range_fold(BT :: lsm_btree_reader:read_file(),
-spec do_range_fold(BT :: hanoi_reader:read_file(),
WorkerPID :: pid(),
SelfOrRef :: pid() | reference(),
Range :: #btree_range{} ) -> ok.
do_range_fold(BT, WorkerPID, SelfOrRef, Range) ->
case lsm_btree_reader:range_fold(fun(Key,Value,_) ->
case hanoi_reader:range_fold(fun(Key,Value,_) ->
WorkerPID ! {level_result, SelfOrRef, Key, Value},
ok
end,

48
src/lsm_btree_merger.erl → src/hanoi_merger.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_merger).
-module(hanoi_merger).
-author('Kresten Krab Thorup <krab@trifork.com>').
%%
@ -31,7 +31,7 @@
-export([merge/5]).
-include("lsm_btree.hrl").
-include("hanoi.hrl").
%%
%% Most likely, there will be plenty of I/O being generated by
@ -41,29 +41,29 @@
-define(LOCAL_WRITER, true).
merge(A,B,C, Size, IsLastLevel) ->
{ok, BT1} = lsm_btree_reader:open(A, sequential),
{ok, BT2} = lsm_btree_reader:open(B, sequential),
{ok, BT1} = hanoi_reader:open(A, sequential),
{ok, BT2} = hanoi_reader:open(B, sequential),
case ?LOCAL_WRITER of
true ->
{ok, Out} = lsm_btree_writer:init([C, Size]);
{ok, Out} = hanoi_writer:init([C, Size]);
false ->
{ok, Out} = lsm_btree_writer:open(C, Size)
{ok, Out} = hanoi_writer:open(C, Size)
end,
{node, AKVs} = lsm_btree_reader:first_node(BT1),
{node, BKVs} = lsm_btree_reader:first_node(BT2),
{node, AKVs} = hanoi_reader:first_node(BT1),
{node, BKVs} = hanoi_reader:first_node(BT2),
{ok, Count, Out2} = scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, 0, {0, none}),
%% finish stream tree
ok = lsm_btree_reader:close(BT1),
ok = lsm_btree_reader:close(BT2),
ok = hanoi_reader:close(BT1),
ok = hanoi_reader:close(BT2),
case ?LOCAL_WRITER of
true ->
{stop, normal, ok, _} = lsm_btree_writer:handle_call(close, self(), Out2);
{stop, normal, ok, _} = hanoi_writer:handle_call(close, self(), Out2);
false ->
ok = lsm_btree_writer:close(Out2)
ok = hanoi_writer:close(Out2)
end,
{ok, Count}.
@ -88,7 +88,7 @@ scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, {0, FromPID}) ->
end;
scan(BT1, BT2, Out, IsLastLevel, [], BKVs, Count, Step) ->
case lsm_btree_reader:next_node(BT1) of
case hanoi_reader:next_node(BT1) of
{node, AKVs} ->
scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, Step);
end_of_data ->
@ -96,7 +96,7 @@ scan(BT1, BT2, Out, IsLastLevel, [], BKVs, Count, Step) ->
end;
scan(BT1, BT2, Out, IsLastLevel, AKVs, [], Count, Step) ->
case lsm_btree_reader:next_node(BT2) of
case hanoi_reader:next_node(BT2) of
{node, BKVs} ->
scan(BT1, BT2, Out, IsLastLevel, AKVs, BKVs, Count, Step);
end_of_data ->
@ -107,9 +107,9 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
if Key1 < Key2 ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key1, Value1}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key1, Value1}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key1, Value1)
ok = hanoi_writer:add(Out2=Out, Key1, Value1)
end,
scan(BT1, BT2, Out2, IsLastLevel, AT, BKVs, Count+1, step(Step));
@ -117,9 +117,9 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
Key2 < Key1 ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key2, Value2}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key2, Value2}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key2, Value2)
ok = hanoi_writer:add(Out2=Out, Key2, Value2)
end,
scan(BT1, BT2, Out2, IsLastLevel, AKVs, BT, Count+1, step(Step));
@ -129,15 +129,15 @@ scan(BT1, BT2, Out, IsLastLevel, [{Key1,Value1}|AT]=AKVs, [{Key2,Value2}|BT]=BKV
true ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key2, Value2}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key2, Value2}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key2, Value2)
ok = hanoi_writer:add(Out2=Out, Key2, Value2)
end,
scan(BT1, BT2, Out2, IsLastLevel, AT, BT, Count+1, step(Step))
end.
scan_only(BT, Out, IsLastLevel, [], Count, Step) ->
case lsm_btree_reader:next_node(BT) of
case hanoi_reader:next_node(BT) of
{node, KVs} ->
scan_only(BT, Out, IsLastLevel, KVs, Count, step(Step));
end_of_data ->
@ -150,8 +150,8 @@ scan_only(BT, Out, true, [{_,?TOMBSTONE}|Rest], Count, Step) ->
scan_only(BT, Out, IsLastLevel, [{Key,Value}|Rest], Count, Step) ->
case ?LOCAL_WRITER of
true ->
{noreply, Out2} = lsm_btree_writer:handle_cast({add, Key, Value}, Out);
{noreply, Out2} = hanoi_writer:handle_cast({add, Key, Value}, Out);
false ->
ok = lsm_btree_writer:add(Out2=Out, Key, Value)
ok = hanoi_writer:add(Out2=Out, Key, Value)
end,
scan_only(BT, Out2, IsLastLevel, Rest, Count+1, step(Step)).

22
src/lsm_btree_nursery.erl → src/hanoi_nursery.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,14 +22,14 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_nursery).
-module(hanoi_nursery).
-author('Kresten Krab Thorup <krab@trifork.com>').
-export([new/1, recover/2, add/3, finish/2, lookup/2, add_maybe_flush/4]).
-export([do_level_fold/3]).
-include("include/lsm_btree.hrl").
-include("lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("hanoi.hrl").
-include_lib("kernel/include/file.hrl").
-record(nursery, { log_file, dir, cache, total_size=0, count=0 }).
@ -147,14 +147,14 @@ finish(#nursery{ dir=Dir, cache=Cache, log_file=LogFile, total_size=_TotalSize,
N when N>0 ->
%% next, flush cache to a new BTree
BTreeFileName = filename:join(Dir, "nursery.data"),
{ok, BT} = lsm_btree_writer:open(BTreeFileName, ?BTREE_SIZE(?TOP_LEVEL)),
{ok, BT} = hanoi_writer:open(BTreeFileName, ?BTREE_SIZE(?TOP_LEVEL)),
try
lists:foreach( fun({Key,Value}) ->
ok = lsm_btree_writer:add(BT, Key, Value)
ok = hanoi_writer:add(BT, Key, Value)
end,
gb_trees:to_list(Cache))
after
ok = lsm_btree_writer:close(BT)
ok = hanoi_writer:close(BT)
end,
% {ok, FileInfo} = file:read_file_info(BTreeFileName),
@ -162,11 +162,11 @@ finish(#nursery{ dir=Dir, cache=Cache, log_file=LogFile, total_size=_TotalSize,
% [ gb_trees:size(Cache), TotalSize, FileInfo#file_info.size ]),
%% inject the B-Tree (blocking RPC)
ok = lsm_btree_level:inject(TopLevel, BTreeFileName),
ok = hanoi_level:inject(TopLevel, BTreeFileName),
%% issue some work if this is a top-level inject (blocks until previous such
%% incremental merge is finished).
lsm_btree_level:incremental_merge(TopLevel, ?BTREE_SIZE(?TOP_LEVEL)),
hanoi_level:incremental_merge(TopLevel, ?BTREE_SIZE(?TOP_LEVEL)),
ok;
_ ->
@ -183,9 +183,9 @@ add_maybe_flush(Key, Value, Nursery=#nursery{ dir=Dir }, Top) ->
{ok, _} = OK ->
OK;
{full, Nursery2} ->
ok = lsm_btree_nursery:finish(Nursery2, Top),
ok = hanoi_nursery:finish(Nursery2, Top),
{error, enoent} = file:read_file_info( filename:join(Dir, "nursery.log")),
lsm_btree_nursery:new(Dir)
hanoi_nursery:new(Dir)
end.
do_level_fold(#nursery{ cache=Cache }, FoldWorkerPID, KeyRange) ->

14
src/lsm_btree_reader.erl → src/hanoi_reader.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,12 +22,12 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_reader).
-module(hanoi_reader).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include_lib("kernel/include/file.hrl").
-include("include/lsm_btree.hrl").
-include("lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("hanoi.hrl").
-export([open/1, open/2,close/1,lookup/2,fold/3,range_fold/4]).
-export([first_node/1,next_node/1]).
@ -232,7 +232,7 @@ find(_, _) ->
read_node(File,{Pos,Size}) ->
% error_logger:info_msg("read_node ~p ~p ~p~n", [File, Pos, Size]),
{ok, <<_:32, Level:16/unsigned, Data/binary>>} = file:pread(File, Pos, Size),
lsm_btree_util:decode_index_node(Level, Data);
hanoi_util:decode_index_node(Level, Data);
read_node(File,Pos) ->
{ok, Pos} = file:position(File, Pos),
@ -246,7 +246,7 @@ read_node(File) ->
0 -> eof;
_ ->
{ok, Data} = file:read(File, Len-2),
{ok, Node} = lsm_btree_util:decode_index_node(Level, Data),
{ok, Node} = hanoi_util:decode_index_node(Level, Data),
{ok, Node}
end.
@ -257,7 +257,7 @@ next_leaf_node(File) ->
eof;
{ok, <<Len:32, 0:16>>} ->
{ok, Data} = file:read(File, Len-2),
lsm_btree_util:decode_index_node(0, Data);
hanoi_util:decode_index_node(0, Data);
{ok, <<Len:32, _:16>>} ->
{ok, _} = file:position(File, {cur,Len-2}),
next_leaf_node(File)

4
src/lsm_btree_sup.erl → src/hanoi_sup.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_sup).
-module(hanoi_sup).
-author('Kresten Krab Thorup <krab@trifork.com>').
-behaviour(supervisor).

6
src/lsm_btree_temp_riak_kv_backend.erl → src/hanoi_temp_riak_kv_backend.erl
View file

@ -20,14 +20,14 @@
%%% NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE
%%%
%%% This is a temporary copy of riak_kv_backend, just here to keep lsm_btree
%%% development private for now. When riak_kv_lsm_btree_backend is moved to
%%% This is a temporary copy of riak_kv_backend, just here to keep hanoi
%%% development private for now. When riak_kv_hanoi_backend is moved to
%%% riak_kv, delete this file.
%%%
%%% NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE NOTE
-module(lsm_btree_temp_riak_kv_backend).
-module(hanoi_temp_riak_kv_backend).
-export([behaviour_info/1]).
-export([callback_after/3]).

4
src/lsm_btree_util.erl → src/hanoi_util.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_util).
-module(hanoi_util).
-author('Kresten Krab Thorup <krab@trifork.com>').
-compile(export_all).

16
src/lsm_btree_writer.erl → src/hanoi_writer.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,10 +22,10 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_writer).
-module(hanoi_writer).
-author('Kresten Krab Thorup <krab@trifork.com>').
-include("lsm_btree.hrl").
-include("hanoi.hrl").
%%
%% Streaming btree writer. Accepts only monotonically increasing keys for put.
@ -80,7 +80,7 @@ init([Name,Size]) ->
% io:format("got name: ~p~n", [Name]),
case file:open( lsm_btree_util:index_file_name(Name),
case file:open( hanoi_util:index_file_name(Name),
[raw, exclusive, write, {delayed_write, 512 * 1024, 2000}]) of
{ok, IdxFile} ->
{ok, BloomFilter} = ebloom:new(erlang:min(Size,16#ffffffff), 0.01, 123),
@ -89,7 +89,7 @@ init([Name,Size]) ->
bloom = BloomFilter
}};
{error, _}=Error ->
error_logger:error_msg("lsm_btree_writer cannot open ~p: ~p~n", [Name, Error]),
error_logger:error_msg("hanoi_writer cannot open ~p: ~p~n", [Name, Error]),
{stop, Error}
end.
@ -113,7 +113,7 @@ terminate(normal,_State) ->
%% premature delete -> cleanup
terminate(_Reason,State) ->
file:close( State#state.index_file ),
file:delete( lsm_btree_util:index_file_name(State#state.name) ).
file:delete( hanoi_util:index_file_name(State#state.name) ).
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
@ -162,7 +162,7 @@ add_record(Level, Key, Value,
end
end,
NewSize = NodeSize + lsm_btree_util:estimate_node_size_increment(List, Key, Value),
NewSize = NodeSize + hanoi_util:estimate_node_size_increment(List, Key, Value),
ebloom:insert( State#state.bloom, Key ),
@ -182,7 +182,7 @@ add_record(Level, Key, Value, #state{ nodes=Nodes }=State) ->
close_node(#state{nodes=[#node{ level=Level, members=NodeMembers }|RestNodes]} = State) ->
OrderedMembers = lists:reverse(NodeMembers),
{ok, DataSize, Data} = lsm_btree_util:encode_index_node(Level, OrderedMembers),
{ok, DataSize, Data} = hanoi_util:encode_index_node(Level, OrderedMembers),
NodePos = State#state.index_file_pos,
ok = file:write(State#state.index_file, Data),

68
src/riak_kv_lsm_btree_backend.erl → src/riak_kv_hanoi_backend.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2012 (c) Basho Technologies, Inc. All Rights Reserved.
%% http://basho.com/ info@basho.com
@ -19,8 +19,8 @@
%%
%% ----------------------------------------------------------------------------
-module(riak_kv_lsm_btree_backend).
-behavior(lsm_btree_temp_riak_kv_backend).
-module(riak_kv_hanoi_backend).
-behavior(hanoi_temp_riak_kv_backend).
-author('Steve Vinoski <steve@basho.com>').
-author('Greg Burd <greg@basho.com>').
@ -45,7 +45,7 @@
-include_lib("eunit/include/eunit.hrl").
-endif.
-include("include/lsm_btree.hrl").
-include("include/hanoi.hrl").
-define(API_VERSION, 1).
%% TODO: for when this backend supports 2i
@ -78,37 +78,37 @@ capabilities(_) ->
capabilities(_, _) ->
{ok, ?CAPABILITIES}.
%% @doc Start the lsm_btree backend
%% @doc Start the hanoi backend
-spec start(integer(), config()) -> {ok, state()} | {error, term()}.
start(Partition, Config) ->
%% Get the data root directory
case app_helper:get_prop_or_env(data_root, Config, lsm_btree) of
case app_helper:get_prop_or_env(data_root, Config, hanoi) of
undefined ->
lager:error("Failed to create lsm_btree dir: data_root is not set"),
lager:error("Failed to create hanoi dir: data_root is not set"),
{error, data_root_unset};
DataRoot ->
AppStart = case application:start(lsm_btree) of
AppStart = case application:start(hanoi) of
ok ->
ok;
{error, {already_started, _}} ->
ok;
{error, StartReason} ->
lager:error("Failed to init the lsm_btree backend: ~p", [StartReason]),
lager:error("Failed to init the hanoi backend: ~p", [StartReason]),
{error, StartReason}
end,
case AppStart of
ok ->
case get_data_dir(DataRoot, integer_to_list(Partition)) of
{ok, DataDir} ->
case lsm_btree:open(DataDir) of
case hanoi:open(DataDir) of
{ok, Tree} ->
{ok, #state{tree=Tree, partition=Partition}};
{error, OpenReason}=OpenError ->
lager:error("Failed to open lsm_btree: ~p\n", [OpenReason]),
lager:error("Failed to open hanoi: ~p\n", [OpenReason]),
OpenError
end;
{error, Reason} ->
lager:error("Failed to start lsm_btree backend: ~p\n", [Reason]),
lager:error("Failed to start hanoi backend: ~p\n", [Reason]),
{error, Reason}
end;
Error ->
@ -116,19 +116,19 @@ start(Partition, Config) ->
end
end.
%% @doc Stop the lsm_btree backend
%% @doc Stop the hanoi backend
-spec stop(state()) -> ok.
stop(#state{tree=Tree}) ->
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
%% @doc Retrieve an object from the lsm_btree backend
%% @doc Retrieve an object from the hanoi backend
-spec get(riak_object:bucket(), riak_object:key(), state()) ->
{ok, any(), state()} |
{ok, not_found, state()} |
{error, term(), state()}.
get(Bucket, Key, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
case lsm_btree:get(Tree, BKey) of
case hanoi:get(Tree, BKey) of
{ok, Value} ->
{ok, Value, State};
not_found ->
@ -137,23 +137,23 @@ get(Bucket, Key, #state{tree=Tree}=State) ->
{error, Reason, State}
end.
%% @doc Insert an object into the lsm_btree backend.
%% @doc Insert an object into the hanoi backend.
-type index_spec() :: {add, Index, SecondaryKey} | {remove, Index, SecondaryKey}.
-spec put(riak_object:bucket(), riak_object:key(), [index_spec()], binary(), state()) ->
{ok, state()} |
{error, term(), state()}.
put(Bucket, Key, _IndexSpecs, Val, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
ok = lsm_btree:put(Tree, BKey, Val),
ok = hanoi:put(Tree, BKey, Val),
{ok, State}.
%% @doc Delete an object from the lsm_btree backend
%% @doc Delete an object from the hanoi backend
-spec delete(riak_object:bucket(), riak_object:key(), [index_spec()], state()) ->
{ok, state()} |
{error, term(), state()}.
delete(Bucket, Key, _IndexSpecs, #state{tree=Tree}=State) ->
BKey = to_object_key(Bucket, Key),
case lsm_btree:delete(Tree, BKey) of
case hanoi:delete(Tree, BKey) of
ok ->
{ok, State};
{error, Reason} ->
@ -170,7 +170,7 @@ fold_buckets(FoldBucketsFun, Acc, Opts, #state{tree=Tree}) ->
BucketFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, {Acc, []}, #btree_range{})
hanoi:sync_fold_range(Tree, FoldFun, {Acc, []}, #btree_range{})
catch
{break, AccFinal} ->
AccFinal
@ -213,7 +213,7 @@ fold_keys(FoldKeysFun, Acc, Opts, #state{tree=Tree}) ->
KeyFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, Acc, Range)
hanoi:sync_fold_range(Tree, FoldFun, Acc, Range)
catch
{break, AccFinal} ->
AccFinal
@ -250,7 +250,7 @@ fold_objects(FoldObjectsFun, Acc, Opts, #state{tree=Tree}) ->
ObjectFolder =
fun() ->
try
lsm_btree:sync_fold_range(Tree, FoldFun, Acc, bucket_range(Bucket))
hanoi:sync_fold_range(Tree, FoldFun, Acc, bucket_range(Bucket))
catch
{break, AccFinal} ->
AccFinal
@ -263,26 +263,26 @@ fold_objects(FoldObjectsFun, Acc, Opts, #state{tree=Tree}) ->
{ok, ObjectFolder()}
end.
%% @doc Delete all objects from this lsm_btree backend
%% @doc Delete all objects from this hanoi backend
-spec drop(state()) -> {ok, state()} | {error, term(), state()}.
drop(#state{}=State) ->
%% TODO: not yet implemented
{ok, State}.
%% @doc Returns true if this lsm_btree backend contains any
%% @doc Returns true if this hanoi backend contains any
%% non-tombstone values; otherwise returns false.
-spec is_empty(state()) -> boolean().
is_empty(#state{tree=Tree}) ->
FoldFun = fun(_K, _V, _Acc) -> throw(ok) end,
try
Range = #btree_range{},
[] =:= lsm_btree:sync_fold_range(Tree, FoldFun, [], Range)
[] =:= hanoi:sync_fold_range(Tree, FoldFun, [], Range)
catch
_:ok ->
false
end.
%% @doc Get the status information for this lsm_btree backend
%% @doc Get the status information for this hanoi backend
-spec status(state()) -> [{atom(), term()}].
status(#state{}) ->
%% TODO: not yet implemented
@ -306,7 +306,7 @@ get_data_dir(DataRoot, Partition) ->
ok ->
{ok, PartitionDir};
{error, Reason} ->
lager:error("Failed to create lsm_btree dir ~s: ~p", [PartitionDir, Reason]),
lager:error("Failed to create hanoi dir ~s: ~p", [PartitionDir, Reason]),
{error, Reason}
end.
@ -377,13 +377,13 @@ from_object_key(LKey) ->
-ifdef(TEST).
simple_test_() ->
?assertCmd("rm -rf test/lsm-btree-backend"),
application:set_env(lsm_btree, data_root, "test/lsm-btree-backend"),
lsm_btree_temp_riak_kv_backend:standard_test(?MODULE, []).
?assertCmd("rm -rf test/hanoi-backend"),
application:set_env(hanoi, data_root, "test/hanoid-backend"),
hanoi_temp_riak_kv_backend:standard_test(?MODULE, []).
custom_config_test_() ->
?assertCmd("rm -rf test/lsm_btree-backend"),
application:set_env(lsm_btree, data_root, ""),
lsm_btree_temp_riak_kv_backend:standard_test(?MODULE, [{data_root, "test/lsm-btree-backend"}]).
?assertCmd("rm -rf test/hanoi-backend"),
application:set_env(hanoi, data_root, ""),
hanoi_temp_riak_kv_backend:standard_test(?MODULE, [{data_root, "test/hanoi-backend"}]).
-endif.

22
test/lsm_btree_drv.erl → test/hanoi_drv.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,8 +22,8 @@
%%
%% ----------------------------------------------------------------------------
%% @Doc Drive a set of lsm BTrees
-module(lsm_btree_drv).
%% @Doc Drive a set of LSM BTrees
-module(hanoi_drv).
-behaviour(gen_server).
@ -90,7 +90,7 @@ init([]) ->
{ok, #state{}}.
handle_call({open, N}, _, #state { btrees = D} = State) ->
case lsm_btree:open(N) of
case hanoi:open(N) of
{ok, Tree} ->
{reply, ok, State#state { btrees = dict:store(N, Tree, D)}};
Otherwise ->
@ -98,7 +98,7 @@ handle_call({open, N}, _, #state { btrees = D} = State) ->
end;
handle_call({close, N}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
case lsm_btree:close(Tree) of
case hanoi:close(Tree) of
ok ->
{reply, ok, State#state { btrees = dict:erase(N, D)}};
Otherwise ->
@ -107,18 +107,18 @@ handle_call({close, N}, _, #state { btrees = D} = State) ->
handle_call({sync_range, Name, Range}, _From,
#state { btrees = D} = State) ->
Tree = dict:fetch(Name, D),
{ok, Ref} = lsm_btree:sync_range(Tree, Range),
{ok, Ref} = hanoi:sync_range(Tree, Range),
Result = sync_range_gather(Ref),
{reply, Result, State};
handle_call({sync_fold_range, Name, Fun, Acc0, Range},
_From,
#state { btrees = D } = State) ->
Tree = dict:fetch(Name, D),
Result = lsm_btree:sync_fold_range(Tree, Fun, Acc0, Range),
Result = hanoi:sync_fold_range(Tree, Fun, Acc0, Range),
{reply, Result, State};
handle_call({put, N, K, V}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
case lsm_btree:put(Tree, K, V) of
case hanoi:put(Tree, K, V) of
ok ->
{reply, ok, State};
Other ->
@ -126,14 +126,14 @@ handle_call({put, N, K, V}, _, #state { btrees = D} = State) ->
end;
handle_call({delete_exist, N, K}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
Reply = lsm_btree:delete(Tree, K),
Reply = hanoi:delete(Tree, K),
{reply, Reply, State};
handle_call({get, N, K}, _, #state { btrees = D} = State) ->
Tree = dict:fetch(N, D),
Reply = lsm_btree:get(Tree, K),
Reply = hanoi:get(Tree, K),
{reply, Reply, State};
handle_call(stop, _, #state{ btrees = D } = State ) ->
[ lsm_btree:close(Tree) || {_,Tree} <- dict:to_list(D) ],
[ hanoi:close(Tree) || {_,Tree} <- dict:to_list(D) ],
{stop, normal, ok, State};
handle_call(_Request, _From, State) ->
Reply = ok,

18
test/lsm_btree_merger_tests.erl → test/hanoi_merger_tests.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_merger_tests).
-module(hanoi_merger_tests).
-ifdef(TEST).
-include_lib("proper/include/proper.hrl").
@ -37,26 +37,26 @@ merge_test() ->
file:delete("test2"),
file:delete("test3"),
{ok, BT1} = lsm_btree_writer:open("test1"),
{ok, BT1} = hanoi_writer:open("test1"),
lists:foldl(fun(N,_) ->
ok = lsm_btree_writer:add(BT1, <<N:128>>, <<"data",N:128>>)
ok = hanoi_writer:add(BT1, <<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(1,10000,2)),
ok = lsm_btree_writer:close(BT1),
ok = hanoi_writer:close(BT1),
{ok, BT2} = lsm_btree_writer:open("test2"),
{ok, BT2} = hanoi_writer:open("test2"),
lists:foldl(fun(N,_) ->
ok = lsm_btree_writer:add(BT2, <<N:128>>, <<"data",N:128>>)
ok = hanoi_writer:add(BT2, <<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(2,5001,1)),
ok = lsm_btree_writer:close(BT2),
ok = hanoi_writer:close(BT2),
self() ! {step, {self(), none}, 2000000000},
{Time,{ok,Count}} = timer:tc(lsm_btree_merger, merge, ["test1", "test2", "test3", 10000, true]),
{Time,{ok,Count}} = timer:tc(hanoi_merger, merge, ["test1", "test2", "test3", 10000, true]),
error_logger:info_msg("time to merge: ~p/sec (time=~p, count=~p)~n", [1000000/(Time/Count), Time/1000000, Count]),

58
test/lsm_btree_tests.erl → test/hanoi_tests.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,10 +22,10 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_tests).
-module(hanoi_tests).
-include("include/lsm_btree.hrl").
-include("src/lsm_btree.hrl").
-include("include/hanoi.hrl").
-include("src/hanoi.hrl").
-ifdef(TEST).
-ifdef(TRIQ).
@ -50,7 +50,7 @@
-record(tree, { elements = dict:new() }).
-record(state, { open = dict:new(),
closed = dict:new() }).
-define(SERVER, lsm_btree_drv).
-define(SERVER, hanoi_drv).
full_test_() ->
{setup,
@ -272,9 +272,9 @@ prop_dict_agree() ->
?FORALL(Cmds, commands(?MODULE),
?TRAPEXIT(
begin
lsm_btree_drv:start_link(),
hanoi_drv:start_link(),
{History,State,Result} = run_commands(?MODULE, Cmds),
lsm_btree_drv:stop(),
hanoi_drv:stop(),
cleanup_test_trees(State),
?WHENFAIL(io:format("History: ~w\nState: ~w\nResult: ~w\n",
[History,State,Result]),
@ -284,64 +284,64 @@ prop_dict_agree() ->
%% UNIT TESTS
%% ----------------------------------------------------------------------
test_tree_simple_1() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<>>, <<"data", 77:128>>),
{ok, <<"data", 77:128>>} = lsm_btree:get(Tree, <<>>),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<>>, <<"data", 77:128>>),
{ok, <<"data", 77:128>>} = hanoi:get(Tree, <<>>),
ok = hanoi:close(Tree).
test_tree_simple_2() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<"ã">>, <<"µ">>),
ok = lsm_btree:delete(Tree, <<"ã">>),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<"ã">>, <<"µ">>),
ok = hanoi:delete(Tree, <<"ã">>),
ok = hanoi:close(Tree).
test_tree_simple_3() ->
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, <<"X">>, <<"Y">>),
{ok, Ref} = lsm_btree:sync_range(Tree, #btree_range{from_key= <<"X">>, to_key= <<"X">>}),
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, <<"X">>, <<"Y">>),
{ok, Ref} = hanoi:sync_range(Tree, #btree_range{from_key= <<"X">>, to_key= <<"X">>}),
?assertEqual(ok,
receive
{fold_done, Ref} -> ok
after 1000 -> {error, timeout}
end),
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
test_tree_simple_4() ->
Key = <<56,11,62,42,35,163,16,100,9,224,8,228,130,94,198,2,126,117,243,
1,122,175,79,159,212,177,30,153,71,91,85,233,41,199,190,58,3,
173,220,9>>,
Value = <<212,167,12,6,105,152,17,80,243>>,
{ok, Tree} = lsm_btree:open("simple"),
ok = lsm_btree:put(Tree, Key, Value),
?assertEqual({ok, Value}, lsm_btree:get(Tree, Key)),
ok = lsm_btree:close(Tree).
{ok, Tree} = hanoi:open("simple"),
ok = hanoi:put(Tree, Key, Value),
?assertEqual({ok, Value}, hanoi:get(Tree, Key)),
ok = hanoi:close(Tree).
test_tree() ->
{ok, Tree} = lsm_btree:open("simple2"),
{ok, Tree} = hanoi:open("simple2"),
lists:foldl(fun(N,_) ->
ok = lsm_btree:put(Tree,
ok = hanoi:put(Tree,
<<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(2,100000,1)),
lists:foldl(fun(N,_) ->
ok = lsm_btree:put(Tree,
ok = hanoi:put(Tree,
<<N:128>>, <<"data",N:128>>)
end,
ok,
lists:seq(4000,6000,1)),
lsm_btree:delete(Tree, <<1500:128>>),
hanoi:delete(Tree, <<1500:128>>),
{Time,{ok,Count}} = timer:tc(?MODULE, run_fold, [Tree,1000,2000]),
error_logger:info_msg("time to fold: ~p/sec (time=~p, count=~p)~n", [1000000/(Time/Count), Time/1000000, Count]),
ok = lsm_btree:close(Tree).
ok = hanoi:close(Tree).
run_fold(Tree,From,To) ->
{ok, PID} = lsm_btree:sync_range(Tree, #btree_range{from_key= <<From:128>>, to_key= <<(To+1):128>>}),
{ok, PID} = hanoi:sync_range(Tree, #btree_range{from_key= <<From:128>>, to_key= <<(To+1):128>>}),
lists:foreach(fun(1500) -> ok;
(N) ->
receive

32
test/lsm_btree_writer_tests.erl → test/hanoi_writer_tests.erl
View file

@ -1,6 +1,6 @@
%% ----------------------------------------------------------------------------
%%
%% lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%% hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
%%
%% Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
%% http://trifork.com/ info@trifork.com
@ -22,7 +22,7 @@
%%
%% ----------------------------------------------------------------------------
-module(lsm_btree_writer_tests).
-module(hanoi_writer_tests).
-ifdef(TEST).
-include_lib("proper/include/proper.hrl").
@ -33,21 +33,21 @@
simple_test() ->
{ok, BT} = lsm_btree_writer:open("testdata"),
ok = lsm_btree_writer:add(BT, <<"A">>, <<"Avalue">>),
ok = lsm_btree_writer:add(BT, <<"B">>, <<"Bvalue">>),
ok = lsm_btree_writer:close(BT),
{ok, BT} = hanoi_writer:open("testdata"),
ok = hanoi_writer:add(BT, <<"A">>, <<"Avalue">>),
ok = hanoi_writer:add(BT, <<"B">>, <<"Bvalue">>),
ok = hanoi_writer:close(BT),
{ok, IN} = lsm_btree_reader:open("testdata"),
{ok, <<"Avalue">>} = lsm_btree_reader:lookup(IN, <<"A">>),
ok = lsm_btree_reader:close(IN),
{ok, IN} = hanoi_reader:open("testdata"),
{ok, <<"Avalue">>} = hanoi_reader:lookup(IN, <<"A">>),
ok = hanoi_reader:close(IN),
ok = file:delete("testdata").
simple1_test() ->
{ok, BT} = lsm_btree_writer:open("testdata"),
{ok, BT} = hanoi_writer:open("testdata"),
Max = 30*1024,
Seq = lists:seq(0, Max),
@ -56,21 +56,21 @@ simple1_test() ->
fun() ->
lists:foreach(
fun(Int) ->
ok = lsm_btree_writer:add(BT, <<Int:128>>, <<"valuevalue/", Int:128>>)
ok = hanoi_writer:add(BT, <<Int:128>>, <<"valuevalue/", Int:128>>)
end,
Seq),
ok = lsm_btree_writer:close(BT)
ok = hanoi_writer:close(BT)
end,
[]),
error_logger:info_msg("time to insert: ~p/sec~n", [1000000/(Time1/Max)]),
{ok, IN} = lsm_btree_reader:open("testdata"),
{ok, <<"valuevalue/", 2048:128>>} = lsm_btree_reader:lookup(IN, <<2048:128>>),
{ok, IN} = hanoi_reader:open("testdata"),
{ok, <<"valuevalue/", 2048:128>>} = hanoi_reader:lookup(IN, <<2048:128>>),
{Time2,Count} = timer:tc(
fun() -> lsm_btree_reader:fold(fun(Key, <<"valuevalue/", Key/binary>>, N) ->
fun() -> hanoi_reader:fold(fun(Key, <<"valuevalue/", Key/binary>>, N) ->
N+1
end,
0,
@ -83,6 +83,6 @@ simple1_test() ->
Max = Count-1,
ok = lsm_btree_reader:close(IN),
ok = hanoi_reader:close(IN),
ok = file:delete("testdata").

2
levelpresence.sh → visualize-hanoi.sh
View file

@ -2,7 +2,7 @@
## ----------------------------------------------------------------------------
##
## lsm_btree: LSM-trees (Log-Structured Merge Trees) Indexed Storage
## hanoi: LSM-trees (Log-Structured Merge Trees) Indexed Storage
##
## Copyright 2011-2012 (c) Trifork A/S. All Rights Reserved.
## http://trifork.com/ info@trifork.com