lmdb-erl/README.md
2013-05-19 00:27:44 -04:00

4.3 KiB

EMDB

EMDB is a NIF library for the Memory-Mapped Database database, aka. MDB.

The main purpose of this package is to provide an Erlang API for this excellent BTREE implementation. Secondly to build an alternative Riak/KV backend and Riak's AAE feature based on this. Finally it'd be nice to build an ETS-compatible API (ala "lets" for LevelDB) to ease adoption in other places where DETS is just not up to the task.

Requirements

  • Erlang R14B04+
  • GCC 4.2+ or MS VisualStudio 2010+

Build

$ make

API

The following functions were implemented:

  • open/1: equivalent to emdb:open(DirName, 10485760).
  • open/2: equivalent to emdb:open(DirName, 10485760, 0).
  • open/3: creates a new MDB database. This call also re-open an already existing one. Arguments are:
    • DirName: database directory name
    • MapSize: database map size (see map.hrl)
    • EnvFlags: database environment flags (see map.hrl). The possible values are defined in emdb.hrl.
  • close/2: closes the database
  • put/2: inserts Key with value Val into the database. Assumes that the key is not present, 'key_exit' is returned otherwise.
  • get/1: retrieves the value stored with Key in the database.
  • del/1: Removes the key-value with key Key from database.
  • update/2 or upd/2: inserts Key with value Val into the database if the key is not present, otherwise updates Key to value Val.
  • drop/1: deletes all key-value pairs in the database.

Usage

$ make
$ ./start.sh
	%% create a new database
	1> {ok, Handle} = emdb:open("/tmp/emdb1").

	%% insert the key <<"a">> with value <<"1">>
	2> ok = emdb:put(Handle, <<"a">>, <<"1">>).

	%% try to re-insert the same key <<"a">>
	3> key_exist = emdb:put(Handle, <<"a">>, <<"2">>).

	%% add a new key-value pair
	4> ok = emdb:put(Handle, <<"b">>, <<"2">>).

	%% search a non-existing key <<"c">>
	5> none = emdb:get(Handle, <<"c">>).

	%% retrieve the value for key <<"b">>
	6> {ok, <<"2">>} = emdb:get(Handle, <<"b">>).

	%% retrieve the value for key <<"a">>
	7> {ok, <<"1">>} = emdb:get(Handle, <<"a">>).

	%% delete key <<"b">>
	8> ok = emdb:del(Handle, <<"b">>).

	%% search a non-existing key <<"b">>
	9> none = emdb:get(Handle, <<"b">>).

	%% delete a non-existing key <<"z">>
	10> none = emdb:del(Handle, <<"z">>).

	%% ensure key <<"a">>'s value is still <<"1">>
	11> {ok, <<"1">>} = emdb:get(Handle, <<"a">>).

	%% update the value for key <<"a">>
	12> ok = emdb:update(Handle, <<"a">>, <<"7">>).

	%% check the new value for key <<"a">>
	13> {ok, <<"7">>} = emdb:get(Handle, <<"a">>).

	%% delete all key-value pairs in the database
	14> ok = emdb:drop(Handle).

	%% try to retrieve key <<"a">> value
	15> none = emdb:get(Handle, <<"a">>).

	%% close the database
	16> ok = emdb:close(Handle).

	...

	17> q().

Note:

The code below creates a new database with 80GB MapSize, avoids fsync after each commit (for an "ACI" but not "D" database we trade durability for speed) and uses the experimental MDB_FIXEDMAP.

	{ok, Handle} = emdb:open("/tmp/emdb2", 85899345920, ?MDB_NOSYNC bor ?MDB_FIXEDMAP).

Performance

See the microbench against:

  • Google's LevelDB (which is slower and can stall unlike Basho's fork of LevelDB)
  • SQLite3
  • Kyoto TreeDB
  • BerkeleyDB 5.x

MDB performs mmap's the database, so unless your dataset is < 2^32 bytes you'll need to run on a 64-bit arch system.

Supported Operating Systems

Should work on:

  • Linux
  • OSX
  • FreeBSD
  • Windows

TODO

  • Fold over keys and/or values
  • Unit tests
  • PropEr testing
  • Bulk "writing"
  • basho_bench driver
  • EQC, PULSE testing
  • Key expirey
  • renaming
    • emdb -> lmdb
    • emdb.c -> lmdb_nif.c
  • improve stats
  • txn API
  • cursor API
  • config
  • use async_nif affinity
  • riak_kv backend
    • use dups
    • 2i
  • aae alternative

Status

Work in progress, not production quality and not supported by Basho Technologies.

LICENSE

EMDB is Copyright (C) 2012-2013 by Aleph Archives and Basho Technologies, Inc., and released under the OpenLDAP License.