Use async_nif to avoid scheduler issues #1

Merged
gburd merged 2 commits from gsb-async-nif into master 2013-05-19 04:30:02 +00:00
Showing only changes of commit 2e552ad3af - Show all commits

View file

@ -1,15 +1,24 @@
EMDB ==== EMDB is a NIF library for the [Memory-Mapped Database](http://highlandsun.com/hyc/mdb/) database, aka. MDB. The main purpose of this package is to provide a **very fast** Riak [backend](http://wiki.basho.com/Storage-Backends.html). EMDB
====
But this module could also be used as a general key-value store to replace: EMDB is a NIF library for the [Memory-Mapped Database](http://highlandsun.com/hyc/mdb/) database, aka. MDB.
The main purpose of this package is to provide an Erlang API for this excellent BTREE implementation. Secondly to build an alternative Riak/KV [backend](http://wiki.basho.com/Storage-Backends.html) and Riak's AAE feature based on this. Finally it'd be nice to build an ETS-compatible API (ala "lets" for LevelDB) to ease adoption in other places where DETS is just not up to the task.
Requirements
------------
* Erlang R14B04+
* GCC 4.2+ or MS VisualStudio 2010+
Build
-----
$ make
API
---
* [DETS](http://www.erlang.org/doc/man/dets.html)
* TokyoCabinet: [TCERL](http://code.google.com/p/tcerl/)
* [QDBM](http://fallabs.com/qdbm/)
* [Bitcask](https://github.com/basho/bitcask)
* [eLevelDB](https://github.com/basho/eleveldb)
* [BerkleyDB](http://www.oracle.com/technetwork/products/berkeleydb/overview/index.html)
* ... Requirements ------------
* Erlang R14B04+ * GCC 4.2+ or MS VisualStudio 2010+ Build ----- $ make API ---
The following functions were implemented: The following functions were implemented:
* `open/1`: equivalent to `emdb:open(DirName, 10485760)`. * `open/1`: equivalent to `emdb:open(DirName, 10485760)`.
@ -22,13 +31,14 @@ But this module could also be used as a general key-value store to replace:
* `put/2`: inserts Key with value Val into the database. Assumes that the key is not present, 'key_exit' is returned otherwise. * `put/2`: inserts Key with value Val into the database. Assumes that the key is not present, 'key_exit' is returned otherwise.
* `get/1`: retrieves the value stored with Key in the database. * `get/1`: retrieves the value stored with Key in the database.
* `del/1`: Removes the key-value with key Key from database. * `del/1`: Removes the key-value with key Key from database.
* `update/2`: inserts Key with value Val into the database if the key is not present, otherwise updates Key to value Val. * `update/2` or `upd/2`: inserts Key with value Val into the database if the key is not present, otherwise updates Key to value Val.
* `drop/1`: deletes all key-value pairs in the database. * `drop/1`: deletes all key-value pairs in the database.
Usage Usage
----- -----
```
$ make $ make
$ ./start.sh $ ./start.sh
%% create a new database %% create a new database
@ -82,32 +92,32 @@ $ ./start.sh
... ...
17> q(). 17> q().
```
#### Note: #### Note:
The code below creates a new database with **80GB** MapSize, **avoid fsync** The code below creates a new database with **80GB** MapSize, **avoids fsync** after each commit (for an "ACI" but not "D" database we trade durability for speed) and uses the experimental **MDB_FIXEDMAP**.
after each commit (for max speed) and use the experimental **MDB_FIXEDMAP**.
```
{ok, Handle} = emdb:open("/tmp/emdb2", 85899345920, ?MDB_NOSYNC bor ?MDB_FIXEDMAP). {ok, Handle} = emdb:open("/tmp/emdb2", 85899345920, ?MDB_NOSYNC bor ?MDB_FIXEDMAP).
```
Performance Performance
----------- -----------
For maximum speed, this library use only binaries for both keys and values. See the [microbench](http://highlandsun.com/hyc/mdb/microbench/) against:
See the impressive [microbench](http://highlandsun.com/hyc/mdb/microbench/) against:
* Google's LevelDB (which is slower and can stall unlike Basho's fork of LevelDB) * Google's LevelDB (which is slower and can stall unlike Basho's fork of LevelDB)
* SQLite3 * SQLite3
* Kyoto TreeDB * Kyoto TreeDB
* BerkeleyDB 5.x * BerkeleyDB 5.x
MDB performs better on 64-bit arch. MDB performs mmap's the database, so unless your dataset is < 2^32 bytes you'll
need to run on a 64-bit arch system.
Supported Operating Systems Supported Operating Systems
-------------- --------------
Should work on 32/64-bit architectures: Should work on:
* Linux * Linux
* OSX * OSX
@ -124,13 +134,24 @@ TODO
* basho_bench driver * basho_bench driver
* EQC, PULSE testing * EQC, PULSE testing
* Key expirey * Key expirey
* Atomic group commit (for 2i) * renaming
* emdb -> lmdb
Volunteers are always welcome! * emdb.c -> lmdb_nif.c
* improve stats
* txn API
* cursor API
* config
* use async_nif affinity
* riak_kv backend
* use dups
* 2i
* aae alternative
Status Status
------ ------
Work in progress, not production quality and not supported by Basho Technologies.
LICENSE LICENSE
------- -------