380 lines
9.3 KiB
Text
380 lines
9.3 KiB
Text
%% -*- Doc -*-
|
|
%% vim: set syntax=asciidoc:
|
|
@author Joseph Wayne Norton <norton@alum.mit.edu>
|
|
@copyright 2011 by Joseph Wayne Norton
|
|
@title LETS - LevelDB-based Erlang Term Storage
|
|
@doc
|
|
LETS is an alternative Erlang Term Storage using LevelDB as the
|
|
storage implementation. LETS tries to address some bad properties of
|
|
ETS and DETS. ETS is limited by physical memory. DETS is limited by
|
|
a 2 GB file size limitation and does not implement ordered sets. LETS
|
|
has neither of these limitations. Data can also be automatically
|
|
compressed using the Snappy compression library.
|
|
|
|
For testing and comparison purposes, LETS supports three
|
|
implementations:
|
|
|
|
- +drv+ C\+\+ Driver with LevelDB backend _(default)_
|
|
- +nif+ C\+\+ NIF with LevelDB backend
|
|
- +ets+ Erlang ETS backend
|
|
|
|
LETS is not intended to be an exact clone of ETS. The currently
|
|
supported ETS APIs are:
|
|
|
|
- +delete/1+
|
|
- +delete/2+
|
|
- +delete_all_objects/1+ _only ets implementation_
|
|
- +first/1+
|
|
- +foldl/3+
|
|
- +foldr/3+
|
|
- +info/1+ _only a subset of items_
|
|
- +info/2+ _only a subset of items_
|
|
- +insert/2+
|
|
- +insert_new/2+ _only ets implementation_
|
|
- +last/1+
|
|
- +lookup/2+
|
|
- +lookup_element/3+
|
|
- +match/1+
|
|
- +match/2+
|
|
- +match/3+
|
|
- +match_delete/2+
|
|
- +match_object/1+
|
|
- +match_object/2+
|
|
- +match_object/3+
|
|
- +member/2+
|
|
- +new/2+
|
|
- +next/2+
|
|
- +prev/2+
|
|
- +select/1+
|
|
- +select/2+
|
|
- +select/3+
|
|
- +select_count/2+
|
|
- +select_delete/2+
|
|
- +select_reverse/1+
|
|
- +select_reverse/2+
|
|
- +select_reverse/3+
|
|
- +tab2list/1+
|
|
|
|
_This repository is experimental in nature - use at your own risk and
|
|
please contribute if you find LETS useful._
|
|
|
|
== Quick Start Recipe
|
|
|
|
To download and build the lets application in one shot, please follow
|
|
this recipe:
|
|
|
|
------
|
|
$ mkdir working-directory-name
|
|
$ cd working-directory-name
|
|
$ git clone git://github.com/norton/snappy.git snappy
|
|
$ git clone git://github.com/norton/leveldb.git leveldb
|
|
$ git clone git://github.com/norton/lets.git lets
|
|
$ cd lets
|
|
$ ./rebar get-deps
|
|
$ ./rebar clean
|
|
$ ./rebar compile
|
|
------
|
|
|
|
For an alternative recipe with other "features" albeit more complex,
|
|
please read further.
|
|
|
|
== Documentation
|
|
|
|
=== Where should I start?
|
|
|
|
This README is the only bit of documentation right now.
|
|
|
|
The QC (a.k.a. QuickCheck, PropEr, etc.) tests underneath the
|
|
"tests/qc" directory should be helpful for understanding the
|
|
specification and behavior of ETS and LETS. These QC tests also
|
|
illustrate several strategies for testing Erlang Driver-based and
|
|
NIF-based implementations.
|
|
|
|
=== What is ETS and DETS?
|
|
|
|
ETS and DETS are Erlang/OTP\'s standard library modules for Erlang
|
|
term storage. ETS is a memory-based implementation. DETS is a
|
|
disk-based implementation.
|
|
|
|
See http://www.erlang.org/doc/man/ets.html and
|
|
http://www.erlang.org/doc/man/dets.html for further details.
|
|
|
|
=== What is LevelDB?
|
|
|
|
LevelDB is a fast key-value storage library written at Google that
|
|
provides an ordered mapping from string keys to string values.
|
|
|
|
See http://code.google.com/p/leveldb/ for further details.
|
|
|
|
=== What is Snappy?
|
|
|
|
Snappy is a fast compression/decompression library written at Google.
|
|
|
|
See http://code.google.com/p/snappy/ for further details.
|
|
|
|
== To download
|
|
|
|
1. Configure your e-mail and name for Git
|
|
+
|
|
------
|
|
$ git config \--global user.email "you@example.com"
|
|
$ git config \--global user.name "Your Name"
|
|
------
|
|
|
|
2. Install Repo
|
|
+
|
|
------
|
|
$ mkdir -p ~/bin
|
|
$ wget -O - https://github.com/android/tools_repo/raw/master/repo > ~/bin/repo
|
|
$ perl -i.bak -pe 's!git://android.git.kernel.org/tools/repo.git!git://github.com/android/tools_repo.git!;' ~/bin/repo
|
|
$ chmod a+x ~/bin/repo
|
|
------
|
|
+
|
|
CAUTION: Since access to kernel.org has been shutdown due to hackers,
|
|
fetch and replace repo tool with android\'s GitHub repository mirror.
|
|
|
|
3. Create working directory
|
|
+
|
|
------
|
|
$ mkdir working-directory-name
|
|
$ cd working-directory-name
|
|
$ repo init -u git://github.com/norton/manifests.git -m lets-default.xml
|
|
------
|
|
+
|
|
NOTE: Your "Git" identity is needed during the init step. Please
|
|
enter the name and email of your GitHub account if you have one. Team
|
|
members having read-write access are recommended to use "repo init -u
|
|
git@github.com:norton/manifests.git -m lets-default-rw.xml".
|
|
+
|
|
TIP: If you want to checkout the latest development version, please
|
|
append " -b dev" to the repo init command.
|
|
|
|
4. Download Git repositories
|
|
+
|
|
------
|
|
$ cd working-directory-name
|
|
$ repo sync
|
|
------
|
|
|
|
For futher information and help for related tools, please refer to the
|
|
following links:
|
|
|
|
- Erlang - http://www.erlang.org/
|
|
* *R14B03 or newer, R14B04 has been tested most recently*
|
|
- Git - http://git-scm.com/
|
|
* *Git 1.5.4 or newer, Git 1.7.7 has been tested recently*
|
|
* _required for Repo and GitHub_
|
|
- GitHub - https://github.com
|
|
- Python - http://www.python.org
|
|
* *Python 2.4 or newer, Python 2.7.1 has been tested most recently
|
|
(CAUTION: Python 3.x might be too new)*
|
|
* _required for Repo_
|
|
- Rebar - https://github.com/basho/rebar/wiki
|
|
- Repo - http://source.android.com/source/git-repo.html
|
|
|
|
== To build - basic recipe
|
|
|
|
1. Get and install an erlang system http://www.erlang.org
|
|
|
|
2. Build
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make compile
|
|
------
|
|
|
|
== To build - optional features
|
|
|
|
A. Dialyzer Testing _basic recipe_
|
|
1. Build Dialyzer\'s PLT _(required once)_
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make build-plt
|
|
------
|
|
+
|
|
TIP: Check Makefile and dialyzer\'s documentation for further
|
|
information.
|
|
|
|
2. Dialyze with specs
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make dialyze
|
|
------
|
|
+
|
|
CAUTION: If you manually run dialyzer with the "-r" option, execute
|
|
"make clean compile" first to avoid finding duplicate beam files
|
|
underneath rebar\'s .eunit directory. Check Makefile for further
|
|
information.
|
|
|
|
3. Dialyze without specs
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make dialyze-nospec
|
|
------
|
|
|
|
== To test - QuickCheck
|
|
|
|
1. Make sure QuickCheck is in your Erlang code path. One simple way
|
|
to accomplish this is by adding the code path to your +~/.erlang+
|
|
resource file.
|
|
+
|
|
------
|
|
true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/quviq/eqc-X.Y.Z/ebin").
|
|
------
|
|
|
|
2. Compile for QuickCheck
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make clean
|
|
$ make compile-eqc eqc-compile
|
|
------
|
|
|
|
3. Run 5,000 QuickCheck tests
|
|
+
|
|
------
|
|
$ cd working-directory-name/src/lib/lets/.eunit
|
|
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin
|
|
|
|
1> qc_statem_lets:qc_run(5000).
|
|
....
|
|
OK, passed 5000 tests
|
|
|
|
9.022% {delete,ok}
|
|
7.800% {new,ok}
|
|
4.535% {match_delete,ok}
|
|
4.491% {lookup,ok}
|
|
4.399% {select,ok}
|
|
4.352% {select_delete,ok}
|
|
4.348% {tab2list,ok}
|
|
4.341% {member,ok}
|
|
4.334% {last,ok}
|
|
4.315% {foldl,ok}
|
|
4.308% {select_reverse,ok}
|
|
4.301% {select_count,ok}
|
|
4.293% {select31,ok}
|
|
4.264% {first,ok}
|
|
4.216% {foldr,ok}
|
|
4.202% {match_object,ok}
|
|
4.184% {match,ok}
|
|
4.056% {insert,ok}
|
|
3.997% {prev,ok}
|
|
3.774% {next,ok}
|
|
3.416% {lookup_element,{error,badarg}}
|
|
1.298% {insert_new,ok}
|
|
0.757% {lookup_element,ok}
|
|
0.516% {next,{error,badarg}}
|
|
0.483% {prev,{error,badarg}}
|
|
true
|
|
.......
|
|
------
|
|
+
|
|
TIP: For testing LevelDB directly using the C bindings, try
|
|
+qc_statemc_lets:qc_run(5000)+.
|
|
|
|
== To test - PropEr
|
|
|
|
1. Make sure PropEr is in your Erlang code path. One simple way to
|
|
accomplish this is by adding the code path to your +~/.erlang+
|
|
resource file.
|
|
+
|
|
------
|
|
true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/proper/ebin").
|
|
------
|
|
|
|
2. Compile for PropEr
|
|
+
|
|
------
|
|
$ cd working-directory-name/src
|
|
$ make clean
|
|
$ make compile-proper proper-compile
|
|
------
|
|
|
|
3. Run 5,000 PropEr tests
|
|
+
|
|
------
|
|
$ cd working-directory-name/src/lib/lets/.eunit
|
|
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin
|
|
|
|
1> qc_statem_lets:qc_run(5000).
|
|
....
|
|
OK: Passed 5000 test(s).
|
|
|
|
11% {new,ok}
|
|
8% {delete,ok}
|
|
4% {member,ok}
|
|
4% {select,ok}
|
|
4% {select_count,ok}
|
|
4% {select_reverse,ok}
|
|
4% {lookup,ok}
|
|
4% {match_object,ok}
|
|
4% {tab2list,ok}
|
|
4% {last,ok}
|
|
4% {match,ok}
|
|
4% {foldl,ok}
|
|
4% {match_delete,ok}
|
|
3% {prev,ok}
|
|
3% {select31,ok}
|
|
3% {select_delete,ok}
|
|
3% {foldr,ok}
|
|
3% {insert,ok}
|
|
3% {first,ok}
|
|
3% {next,ok}
|
|
3% {lookup_element,{error,badarg}}
|
|
1% {insert_new,ok}
|
|
0% {prev,{error,badarg}}
|
|
0% {lookup_element,ok}
|
|
0% {next,{error,badarg}}
|
|
true
|
|
.......
|
|
------
|
|
|
|
== Roadmap
|
|
|
|
- Documentation
|
|
* Explain how to run QuickCheck/PropEr tests using a new rebar
|
|
plugin.
|
|
* Explain how to build and to run lets with valgrind enabled
|
|
OTP/Erlang virtual machine
|
|
|
|
- Testing
|
|
* Functional
|
|
** Update test model to include LevelDB\'s database, read, and
|
|
write options. These options have not been tested.
|
|
** Update test model to include LevelDB\'s destroy and repair
|
|
operations. These operations have not been tested.
|
|
* Performance (TBD)
|
|
* Stability (TBD)
|
|
|
|
- New APIs (TBD)
|
|
* +insert_new/2+
|
|
(http://code.google.com/p/leveldb/issues/detail?id=42)
|
|
|
|
* +delete_all_objects/1+
|
|
(http://code.google.com/p/leveldb/issues/detail?id=43)
|
|
|
|
* Add custom (i.e. not supported by native ETS) APIs for providing
|
|
access to LevelDB\'s iterators for +drv+ and +nif+ backend
|
|
implementations.
|
|
|
|
- Existing APIs (TBD)
|
|
* +delete/1+
|
|
(http://code.google.com/p/leveldb/issues/detail?id=48)
|
|
|
|
* +new/2+ -
|
|
(http://code.google.com/p/leveldb/issues/detail?id=49)
|
|
|
|
* +new/2+ - investigate if LevelDB\'s snapshot feature is useful (or
|
|
not) for LETS
|
|
|
|
* +info/2+ - investigate if LevelDB\'s implementation can (easily)
|
|
support size and memory info items
|
|
|
|
* consider adding explicit read_options and write_options for LET\'s
|
|
operations (rather than just +new/2+, +destroy/2+, and +repair/2+
|
|
operations).
|
|
|
|
@end
|