776 lines
13 KiB
Markdown
776 lines
13 KiB
Markdown
|
|
|
|
#LETS - LevelDB-based Erlang Term Storage#
|
|
|
|
|
|
Copyright (c) 2011 by Joseph Wayne Norton
|
|
|
|
__Authors:__ Joseph Wayne Norton ([`norton@alum.mit.edu`](mailto:norton@alum.mit.edu)).<p>LETS is an alternative Erlang Term Storage using LevelDB as the
|
|
storage implementation. LETS tries to address some bad properties of
|
|
ETS and DETS. ETS is limited by physical memory. DETS is limited by
|
|
a 2 GB file size limitation and does not implement ordered sets. LETS
|
|
has neither of these limitations. Data can also be automatically
|
|
compressed using the Snappy compression library.</p>
|
|
<p>For testing and comparison purposes, LETS supports three
|
|
implementations:</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<tt>drv</tt> C+\+ Driver with LevelDB backend <em>(default)</em>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>nif</tt> C+\+ NIF with LevelDB backend
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>ets</tt> Erlang ETS backend
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
<p>LETS is not intended to be an exact clone of ETS. The currently
|
|
supported ETS APIs are:</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<tt>delete/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>delete/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>delete_all_objects/1</tt> <em>only ets implementation</em>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>first/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>foldl/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>foldr/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>info/1</tt> <em>only a subset of items</em>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>info/2</tt> <em>only a subset of items</em>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>insert/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>insert_new/2</tt> <em>only ets implementation</em>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>last/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>lookup/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>lookup_element/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match_delete/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match_object/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match_object/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>match_object/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>member/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>new/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>next/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>prev/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select_count/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select_delete/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select_reverse/1</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select_reverse/2</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>select_reverse/3</tt>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>tab2list/1</tt>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
<p><em>This repository is experimental in nature - use at your own risk and
|
|
please contribute if you find LETS useful.</em></p>
|
|
|
|
<h2 id="_quick_start_recipe">Quick Start Recipe</h2>
|
|
|
|
<p>To download and build the lets application in one shot, please follow
|
|
this recipe:</p>
|
|
|
|
|
|
<pre><tt>$ mkdir working-directory-name
|
|
$ cd working-directory-name
|
|
$ git clone git://github.com/norton/snappy.git snappy
|
|
$ git clone git://github.com/norton/leveldb.git leveldb
|
|
$ git clone git://github.com/norton/lets.git lets
|
|
$ cd lets
|
|
$ ./rebar get-deps
|
|
$ ./rebar clean
|
|
$ ./rebar compile</tt></pre>
|
|
|
|
<p>For an alternative recipe with other "features" albeit more complex,
|
|
please read further.</p>
|
|
|
|
|
|
|
|
<h2 id="_documentation">Documentation</h2>
|
|
|
|
|
|
<h3 id="_where_should_i_start">Where should I start?</h3>
|
|
<p>This README is the only bit of documentation right now.</p>
|
|
<p>The QC (a.k.a. QuickCheck, PropEr, etc.) tests underneath the
|
|
"tests/qc" directory should be helpful for understanding the
|
|
specification and behavior of ETS and LETS. These QC tests also
|
|
illustrate several strategies for testing Erlang Driver-based and
|
|
NIF-based implementations.</p>
|
|
|
|
|
|
<h3 id="_what_is_ets_and_dets">What is ETS and DETS?</h3>
|
|
<p>ETS and DETS are Erlang/OTP's standard library modules for Erlang
|
|
term storage. ETS is a memory-based implementation. DETS is a
|
|
disk-based implementation.</p>
|
|
<p>See <a href="http://www.erlang.org/doc/man/ets.html">http://www.erlang.org/doc/man/ets.html</a> and
|
|
<a href="http://www.erlang.org/doc/man/dets.html">http://www.erlang.org/doc/man/dets.html</a> for further details.</p>
|
|
|
|
|
|
<h3 id="_what_is_leveldb">What is LevelDB?</h3>
|
|
<p>LevelDB is a fast key-value storage library written at Google that
|
|
provides an ordered mapping from string keys to string values.</p>
|
|
<p>See <a href="http://code.google.com/p/leveldb/">http://code.google.com/p/leveldb/</a> for further details.</p>
|
|
|
|
|
|
<h3 id="_what_is_snappy">What is Snappy?</h3>
|
|
<p>Snappy is a fast compression/decompression library written at Google.</p>
|
|
<p>See <a href="http://code.google.com/p/snappy/">http://code.google.com/p/snappy/</a> for further details.</p>
|
|
|
|
|
|
|
|
|
|
<h2 id="_to_download">To download</h2>
|
|
|
|
<ol class="arabic">
|
|
<li>
|
|
<p>
|
|
Configure your e-mail and name for Git
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ git config \--global user.email "you@example.com"
|
|
$ git config \--global user.name "Your Name"</tt></pre>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Install Repo
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ mkdir -p ~/bin
|
|
$ wget -O - https://github.com/android/tools_repo/raw/master/repo > ~/bin/repo
|
|
$ perl -i.bak -pe 's!git://android.git.kernel.org/tools/repo.git!git://github.com/android/tools_repo.git!;' ~/bin/repo
|
|
$ chmod a+x ~/bin/repo</tt></pre>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Caution
|
|
</td>
|
|
<td class="content">Since access to kernel.org has been shutdown due to hackers,
|
|
fetch and replace repo tool with android's GitHub repository mirror.</td>
|
|
</tr></table>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Create working directory
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ mkdir working-directory-name
|
|
$ cd working-directory-name
|
|
$ repo init -u git://github.com/norton/manifests.git -m lets-default.xml</tt></pre>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Note
|
|
</td>
|
|
<td class="content">Your "Git" identity is needed during the init step. Please
|
|
enter the name and email of your GitHub account if you have one. Team
|
|
members having read-write access are recommended to use "repo init -u
|
|
<a href="mailto:git@github.com">git@github.com</a>:norton/manifests.git -m lets-default-rw.xml".</td>
|
|
</tr></table>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Tip
|
|
</td>
|
|
<td class="content">If you want to checkout the latest development version, please
|
|
append " -b dev" to the repo init command.</td>
|
|
</tr></table>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Download Git repositories
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name
|
|
$ repo sync</tt></pre>
|
|
|
|
</li>
|
|
</ol>
|
|
<p>For futher information and help for related tools, please refer to the
|
|
following links:</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
Erlang - <a href="http://www.erlang.org/">http://www.erlang.org/</a>
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<strong>R14B03 or newer, R14B04 has been tested most recently</strong>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Git - <a href="http://git-scm.com/">http://git-scm.com/</a>
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<strong>Git 1.5.4 or newer, Git 1.7.7 has been tested recently</strong>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<em>required for Repo and GitHub</em>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
GitHub - <a href="https://github.com">https://github.com</a>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Python - <a href="http://www.python.org">http://www.python.org</a>
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<strong>Python 2.4 or newer, Python 2.7.1 has been tested most recently
|
|
(CAUTION: Python 3.x might be too new)</strong>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<em>required for Repo</em>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Rebar - <a href="https://github.com/basho/rebar/wiki">https://github.com/basho/rebar/wiki</a>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Repo - <a href="http://source.android.com/source/git-repo.md">http://source.android.com/source/git-repo.html</a>
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
|
|
|
|
|
|
<h2 id="_to_build_basic_recipe">To build - basic recipe</h2>
|
|
|
|
<ol class="arabic">
|
|
<li>
|
|
<p>
|
|
Get and install an erlang system <a href="http://www.erlang.org">http://www.erlang.org</a>
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Build
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make compile</tt></pre>
|
|
|
|
</li>
|
|
</ol>
|
|
|
|
|
|
|
|
<h2 id="_to_build_optional_features">To build - optional features</h2>
|
|
|
|
<ol class="upperalpha">
|
|
<li>
|
|
<p>
|
|
Dialyzer Testing <em>basic recipe</em>
|
|
</p>
|
|
<ol class="arabic">
|
|
<li>
|
|
<p>
|
|
Build Dialyzer's PLT <em>(required once)</em>
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make build-plt</tt></pre>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Tip
|
|
</td>
|
|
<td class="content">Check Makefile and dialyzer's documentation for further
|
|
information.</td>
|
|
</tr></table>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Dialyze with specs
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make dialyze</tt></pre>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Caution
|
|
</td>
|
|
<td class="content">If you manually run dialyzer with the "-r" option, execute
|
|
"make clean compile" first to avoid finding duplicate beam files
|
|
underneath rebar's .eunit directory. Check Makefile for further
|
|
information.</td>
|
|
</tr></table>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Dialyze without specs
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make dialyze-nospec</tt></pre>
|
|
|
|
</li>
|
|
</ol>
|
|
</li>
|
|
</ol>
|
|
|
|
|
|
|
|
<h2 id="_to_test_quickcheck">To test - QuickCheck</h2>
|
|
|
|
<ol class="arabic">
|
|
<li>
|
|
<p>
|
|
Make sure QuickCheck is in your Erlang code path. One simple way
|
|
to accomplish this is by adding the code path to your <tt>~/.erlang</tt>
|
|
resource file.
|
|
</p>
|
|
|
|
|
|
<pre><tt>true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/quviq/eqc-X.Y.Z/ebin").</tt></pre>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Compile for QuickCheck
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make clean
|
|
$ make compile-eqc eqc-compile</tt></pre>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Run 5,000 QuickCheck tests
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src/lib/lets/.eunit
|
|
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin
|
|
|
|
1> qc_statem_lets:qc_run(5000).
|
|
....
|
|
OK, passed 5000 tests
|
|
|
|
9.022% {delete,ok}
|
|
7.800% {new,ok}
|
|
4.535% {match_delete,ok}
|
|
4.491% {lookup,ok}
|
|
4.399% {select,ok}
|
|
4.352% {select_delete,ok}
|
|
4.348% {tab2list,ok}
|
|
4.341% {member,ok}
|
|
4.334% {last,ok}
|
|
4.315% {foldl,ok}
|
|
4.308% {select_reverse,ok}
|
|
4.301% {select_count,ok}
|
|
4.293% {select31,ok}
|
|
4.264% {first,ok}
|
|
4.216% {foldr,ok}
|
|
4.202% {match_object,ok}
|
|
4.184% {match,ok}
|
|
4.056% {insert,ok}
|
|
3.997% {prev,ok}
|
|
3.774% {next,ok}
|
|
3.416% {lookup_element,{error,badarg}}
|
|
1.298% {insert_new,ok}
|
|
0.757% {lookup_element,ok}
|
|
0.516% {next,{error,badarg}}
|
|
0.483% {prev,{error,badarg}}
|
|
true
|
|
.......</tt></pre>
|
|
|
|
|
|
<table><tr>
|
|
<td class="icon">
|
|
Tip
|
|
</td>
|
|
<td class="content">For testing LevelDB directly using the C bindings, try
|
|
<tt>qc_statemc_lets:qc_run(5000)</tt>.</td>
|
|
</tr></table>
|
|
|
|
</li>
|
|
</ol>
|
|
|
|
|
|
|
|
<h2 id="_to_test_proper">To test - PropEr</h2>
|
|
|
|
<ol class="arabic">
|
|
<li>
|
|
<p>
|
|
Make sure PropEr is in your Erlang code path. One simple way to
|
|
accomplish this is by adding the code path to your <tt>~/.erlang</tt>
|
|
resource file.
|
|
</p>
|
|
|
|
|
|
<pre><tt>true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/proper/ebin").</tt></pre>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Compile for PropEr
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src
|
|
$ make clean
|
|
$ make compile-proper proper-compile</tt></pre>
|
|
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Run 5,000 PropEr tests
|
|
</p>
|
|
|
|
|
|
<pre><tt>$ cd working-directory-name/src/lib/lets/.eunit
|
|
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin
|
|
|
|
1> qc_statem_lets:qc_run(5000).
|
|
....
|
|
OK: Passed 5000 test(s).
|
|
|
|
11% {new,ok}
|
|
8% {delete,ok}
|
|
4% {member,ok}
|
|
4% {select,ok}
|
|
4% {select_count,ok}
|
|
4% {select_reverse,ok}
|
|
4% {lookup,ok}
|
|
4% {match_object,ok}
|
|
4% {tab2list,ok}
|
|
4% {last,ok}
|
|
4% {match,ok}
|
|
4% {foldl,ok}
|
|
4% {match_delete,ok}
|
|
3% {prev,ok}
|
|
3% {select31,ok}
|
|
3% {select_delete,ok}
|
|
3% {foldr,ok}
|
|
3% {insert,ok}
|
|
3% {first,ok}
|
|
3% {next,ok}
|
|
3% {lookup_element,{error,badarg}}
|
|
1% {insert_new,ok}
|
|
0% {prev,{error,badarg}}
|
|
0% {lookup_element,ok}
|
|
0% {next,{error,badarg}}
|
|
true
|
|
.......</tt></pre>
|
|
|
|
</li>
|
|
</ol>
|
|
|
|
|
|
|
|
<h2 id="_roadmap">Roadmap</h2>
|
|
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
Documentation
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
Explain how to run QuickCheck/PropEr tests using a new rebar
|
|
plugin.
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Explain how to build and to run lets with valgrind enabled
|
|
OTP/Erlang virtual machine
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Testing
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
Functional
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
Update test model to include LevelDB's database, read, and
|
|
write options. These options have not been tested.
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Update test model to include LevelDB's destroy and repair
|
|
operations. These operations have not been tested.
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Performance (TBD)
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Stability (TBD)
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
New APIs (TBD)
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<tt>insert_new/2</tt>
|
|
(<a href="http://code.google.com/p/leveldb/issues/detail?id=42">http://code.google.com/p/leveldb/issues/detail?id=42</a>)
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>delete_all_objects/1</tt>
|
|
(<a href="http://code.google.com/p/leveldb/issues/detail?id=43">http://code.google.com/p/leveldb/issues/detail?id=43</a>)
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Add custom (i.e. not supported by native ETS) APIs for providing
|
|
access to LevelDB's iterators for <tt>drv</tt> and <tt>nif</tt> backend
|
|
implementations.
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
Existing APIs (TBD)
|
|
</p>
|
|
<ul>
|
|
<li>
|
|
<p>
|
|
<tt>delete/1</tt>
|
|
(<a href="http://code.google.com/p/leveldb/issues/detail?id=48">http://code.google.com/p/leveldb/issues/detail?id=48</a>)
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>new/2</tt> -
|
|
(<a href="http://code.google.com/p/leveldb/issues/detail?id=49">http://code.google.com/p/leveldb/issues/detail?id=49</a>)
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>new/2</tt> - investigate if LevelDB's snapshot feature is useful (or
|
|
not) for LETS
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
<tt>info/2</tt> - investigate if LevelDB's implementation can (easily)
|
|
support size and memory info items
|
|
</p>
|
|
</li>
|
|
<li>
|
|
<p>
|
|
consider adding explicit read_options and write_options for LET's
|
|
operations (rather than just <tt>new/2</tt>, <tt>destroy/2</tt>, and <tt>repair/2</tt>
|
|
operations).
|
|
</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
|
|
|
|
|
|
|
|
##Modules##
|
|
|
|
|
|
<table width="100%" border="0" summary="list of modules">
|
|
<tr><td><a href="lets.md" class="module">lets</a></td></tr>
|
|
<tr><td><a href="lets_drv.md" class="module">lets_drv</a></td></tr>
|
|
<tr><td><a href="lets_ets.md" class="module">lets_ets</a></td></tr>
|
|
<tr><td><a href="lets_nif.md" class="module">lets_nif</a></td></tr></table>
|
|
|