#LETS - LevelDB-based Erlang Term Storage# Copyright (c) 2011 by Joseph Wayne Norton __Authors:__ Joseph Wayne Norton ([`norton@alum.mit.edu`](mailto:norton@alum.mit.edu)).

LETS is an alternative Erlang Term Storage using LevelDB as the storage implementation. LETS tries to address some bad properties of ETS and DETS. ETS is limited by physical memory. DETS is limited by a 2 GB file size limitation and does not implement ordered sets. LETS has neither of these limitations. Data can also be automatically compressed using the Snappy compression library.

For testing and comparison purposes, LETS supports three implementations:

LETS is not intended to be an exact clone of ETS. The currently supported ETS APIs are:

This repository is experimental in nature - use at your own risk and please contribute if you find LETS useful.

Quick Start Recipe

To download and build the lets application in one shot, please follow this recipe:

$ mkdir working-directory-name
$ cd working-directory-name
$ git clone git://github.com/norton/snappy.git snappy
$ git clone git://github.com/norton/leveldb.git leveldb
$ git clone git://github.com/norton/lets.git lets
$ cd lets
$ ./rebar get-deps
$ ./rebar clean
$ ./rebar compile

For an alternative recipe with other "features" albeit more complex, please read further.

Documentation

Where should I start?

This README is the only bit of documentation right now.

The QC (a.k.a. QuickCheck, PropEr, etc.) tests underneath the "tests/qc" directory should be helpful for understanding the specification and behavior of ETS and LETS. These QC tests also illustrate several strategies for testing Erlang Driver-based and NIF-based implementations.

What is ETS and DETS?

ETS and DETS are Erlang/OTP's standard library modules for Erlang term storage. ETS is a memory-based implementation. DETS is a disk-based implementation.

See http://www.erlang.org/doc/man/ets.html and http://www.erlang.org/doc/man/dets.html for further details.

What is LevelDB?

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

See http://code.google.com/p/leveldb/ for further details.

What is Snappy?

Snappy is a fast compression/decompression library written at Google.

See http://code.google.com/p/snappy/ for further details.

To download

  1. Configure your e-mail and name for Git 

    $ git config \--global user.email "you@example.com"
$ git config \--global user.name "Your Name"

  2. Install Repo 

    $ mkdir -p ~/bin
$ wget -O - https://github.com/android/tools_repo/raw/master/repo > ~/bin/repo
$ perl -i.bak -pe 's!git://android.git.kernel.org/tools/repo.git!git://github.com/android/tools_repo.git!;' ~/bin/repo
$ chmod a+x ~/bin/repo
    Caution Since access to kernel.org has been shutdown due to hackers, fetch and replace repo tool with android's GitHub repository mirror.

  3. Create working directory 

    $ mkdir working-directory-name
$ cd working-directory-name
$ repo init -u git://github.com/norton/manifests.git -m lets-default.xml
    Note Your "Git" identity is needed during the init step. Please enter the name and email of your GitHub account if you have one. Team members having read-write access are recommended to use "repo init -u git@github.com:norton/manifests.git -m lets-default-rw.xml".
    Tip If you want to checkout the latest development version, please append " -b dev" to the repo init command.

  4. Download Git repositories 

    $ cd working-directory-name
$ repo sync

For futher information and help for related tools, please refer to the following links:

To build - basic recipe

  1. Get and install an erlang system http://www.erlang.org

  2. Build 

    $ cd working-directory-name/src
$ make compile

To build - optional features

  1. Dialyzer Testing basic recipe

    1. Build Dialyzer's PLT (required once) 

      $ cd working-directory-name/src
$ make build-plt
      Tip Check Makefile and dialyzer's documentation for further information.

    2. Dialyze with specs 

      $ cd working-directory-name/src
$ make dialyze
      Caution If you manually run dialyzer with the "-r" option, execute "make clean compile" first to avoid finding duplicate beam files underneath rebar's .eunit directory. Check Makefile for further information.

    3. Dialyze without specs 

      $ cd working-directory-name/src
$ make dialyze-nospec

To test - QuickCheck

  1. Make sure QuickCheck is in your Erlang code path. One simple way to accomplish this is by adding the code path to your ~/.erlang resource file. 

    true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/quviq/eqc-X.Y.Z/ebin").

  2. Compile for QuickCheck 

    $ cd working-directory-name/src
$ make clean
$ make compile-eqc eqc-compile

  3. Run 5,000 QuickCheck tests 

    $ cd working-directory-name/src/lib/lets/.eunit
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin

1> qc_statem_lets:qc_run(5000).
....
OK, passed 5000 tests

9.022% {delete,ok}
7.800% {new,ok}
4.535% {match_delete,ok}
4.491% {lookup,ok}
4.399% {select,ok}
4.352% {select_delete,ok}
4.348% {tab2list,ok}
4.341% {member,ok}
4.334% {last,ok}
4.315% {foldl,ok}
4.308% {select_reverse,ok}
4.301% {select_count,ok}
4.293% {select31,ok}
4.264% {first,ok}
4.216% {foldr,ok}
4.202% {match_object,ok}
4.184% {match,ok}
4.056% {insert,ok}
3.997% {prev,ok}
3.774% {next,ok}
3.416% {lookup_element,{error,badarg}}
1.298% {insert_new,ok}
0.757% {lookup_element,ok}
0.516% {next,{error,badarg}}
0.483% {prev,{error,badarg}}
true
.......
    Tip For testing LevelDB directly using the C bindings, try qc_statemc_lets:qc_run(5000).

To test - PropEr

  1. Make sure PropEr is in your Erlang code path. One simple way to accomplish this is by adding the code path to your ~/.erlang resource file. 

    true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/lib/proper/ebin").

  2. Compile for PropEr 

    $ cd working-directory-name/src
$ make clean
$ make compile-proper proper-compile

  3. Run 5,000 PropEr tests 

    $ cd working-directory-name/src/lib/lets/.eunit
$ erl -smp +A 5 -pz ../../sext/ebin -pz ../../qc/ebin

1> qc_statem_lets:qc_run(5000).
....
OK: Passed 5000 test(s).

11% {new,ok}
8% {delete,ok}
4% {member,ok}
4% {select,ok}
4% {select_count,ok}
4% {select_reverse,ok}
4% {lookup,ok}
4% {match_object,ok}
4% {tab2list,ok}
4% {last,ok}
4% {match,ok}
4% {foldl,ok}
4% {match_delete,ok}
3% {prev,ok}
3% {select31,ok}
3% {select_delete,ok}
3% {foldr,ok}
3% {insert,ok}
3% {first,ok}
3% {next,ok}
3% {lookup_element,{error,badarg}}
1% {insert_new,ok}
0% {prev,{error,badarg}}
0% {lookup_element,ok}
0% {next,{error,badarg}}
true
.......

Roadmap

##Modules##
lets
lets_drv
lets_ets
lets_nif