This is a Riak storage backend that uses WiredTiger, an Erlang interface to the WiredTiger k/v pair storage engine, and an abstract async thread-pool layer for Erlang's NIF API called c_src/async_nif.h.
Find a file
2013-07-02 22:23:32 -04:00
c_src Evict older half of items in the cache by removing items from the end of the list, don't waste cycles computing timestamps. 2013-07-02 22:23:32 -04:00
src Changed conditions for worker thread creation. 2013-07-02 16:46:04 -04:00
tools WIP: replcae the kbtree with khash, we don't need the tree features (yet, if ever) and hash is faster; add a most-recently-used stash for contexts as it's highly likely that worker threads will do many operations with the same shape/signature of session/cursors/tables/config and that path can be lock-free as well making it much faster (one would hope); somewhere something is stepping on read-only ErlNifBinary data and so a crc check is failing and causing the runtime to abort, that's the latest item to find/fix. 2013-06-04 14:45:23 -04:00
.gdbinit Move the FIFO Queue implementation into its own file (fifo_q.h). Work 2013-04-18 10:37:36 -04:00
.gitignore Inline the fifo_q functions to speed them up and silence compiler warnings 2013-04-17 11:17:13 -04:00
AUTHORS Opening a connection now involves passing in both a configuration 2013-04-06 17:19:59 -04:00
enable-wterl Remove check for other module. 2013-03-28 10:23:56 -04:00
Makefile WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
README.md Another TODO item. 2013-05-02 11:29:37 -04:00
rebar update rebar and rebar.config 2012-10-26 09:25:35 -04:00
rebar.config WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
update-version.sh Add a longer, multi-table test. Use a release version of WiredTiger from now on. 2013-06-18 09:21:58 -04:00

wterl is an Erlang interface to the WiredTiger database, and is written to support a Riak storage backend that uses WiredTiger.

Remaining work includes:

TODO:

  • Find/fix any code marked "TODO:"
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1 during fold_objects/4?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next/1 during is_empty/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next_value/1 during status/1?
    • Why do we see {error, {ebusy, _}} result on wterl:drop/2?
    • Determine a better way to estimate the number of sessions we should configure WT for at startup in riak_kv_wterl_backend:max_sessions/1.
  • Make sure Erlang is optimizing for selective receive in async_nif_enqueue/3 because in the eLevelDB driver there is a comment: "This cannot be a separate function. Code must be inline to trigger Erlang compiler's use of optimized selective receive."
  • Provide a way to configure the cursor options, right now they are always "raw,overwrite".
  • Add support for Riak/KV 2i indexes using the same design pattern as eLevelDB (in a future version consider alternate schema)
  • If an operation using a shared cursor results in a non-normal error then it should be closed/discarded from the recycled pool
  • Cache cursors based on hash(table/config) rather than just table.
  • Finish NIF unload/reload functions and test.
  • Test an upgrade, include a format/schema/WT change.
  • When WT_PANIC is returned first try to unload/reload then driver and reset all state, if that fails then exit gracefully.
  • Currently the riak_kv_wterl_backend module is stored in this repository, but it really belongs in the riak_kv repository.
  • wterl:truncate/5 can segv, and its tests are commented out
  • Add async_nif and wterl NIF stats to the results provided by the stats API
  • Longer term ideas/changes to consider:
    • More testing, especially pulse/qc
    • Riak/KV integration
      • Store 2i indexes in separate tables
      • Store buckets, in separate tables and keep a <<bucket/key>> index to ensure that folds across a vnode are easy
      • Provide a drop bucket API call
      • Support key expirey
      • An ets API (like the LevelDB's lets project)
      • Use mime-type to inform WT's schema for key value encoding
  • Other use cases within Riak
    • An AAE driver using WT
    • An ability to store the ring file via WT

Deploying

You can deploy wterl into a Riak devrel cluster using the enable-wterl script. Clone the riak repo, change your working directory to it, and then execute the enable-wterl script. It adds wterl as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the WiredTiger storage backend.