greg/wterl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
This is a Riak storage backend that uses WiredTiger, an Erlang interface to the WiredTiger k/v pair storage engine, and an abstract async thread-pool layer for Erlang's NIF API called c_src/async_nif.h.
234 commits 19 branches 0 tags 33 MiB
C 51.3%
Erlang 44.4%
Shell 2.9%
Makefile 1.4%
Find a file
Gregory Burd c41e411a92 Worker threads come and go as needed with a lower bound of 2 and an 
upper bound of ASYNC_NIF_MAX_WORKERS.  Stats were improved to use
thread local storage for measures.  With stats working again wterl
uses them to determine who to evict.  Wterl's signature calculation
for an operation wasn't correct and so the cache wasn't efficient at
all, this has been fixed.
2013-06-25 13:31:43 -04:00
c_src Worker threads come and go as needed with a lower bound of 2 and an 2013-06-25 13:31:43 -04:00
src Minor oversight, fixed. 2013-06-19 14:42:56 -04:00
tools WIP: replcae the kbtree with khash, we don't need the tree features (yet, if ever) and hash is faster; add a most-recently-used stash for contexts as it's highly likely that worker threads will do many operations with the same shape/signature of session/cursors/tables/config and that path can be lock-free as well making it much faster (one would hope); somewhere something is stepping on read-only ErlNifBinary data and so a crc check is failing and causing the runtime to abort, that's the latest item to find/fix. 2013-06-04 14:45:23 -04:00
.gdbinit Move the FIFO Queue implementation into its own file (fifo_q.h). Work 2013-04-18 10:37:36 -04:00
.gitignore Inline the fifo_q functions to speed them up and silence compiler warnings 2013-04-17 11:17:13 -04:00
AUTHORS Opening a connection now involves passing in both a configuration 2013-04-06 17:19:59 -04:00
enable-wterl Remove check for other module. 2013-03-28 10:23:56 -04:00
Makefile WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
README.md Another TODO item. 2013-05-02 11:29:37 -04:00
rebar update rebar and rebar.config 2012-10-26 09:25:35 -04:00
rebar.config WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
update-version.sh Add a longer, multi-table test. Use a release version of WiredTiger from now on. 2013-06-18 09:21:58 -04:00

README.md

wterl is an Erlang interface to the WiredTiger database, and is written to support a Riak storage backend that uses WiredTiger.

Remaining work includes:

TODO:

  • Find/fix any code marked "TODO:"
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1 during fold_objects/4?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_close/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next/1 during is_empty/1?
    • Why do we see {error, {eperm, _}} result on wterl:cursor_next_value/1 during status/1?
    • Why do we see {error, {ebusy, _}} result on wterl:drop/2?
    • Determine a better way to estimate the number of sessions we should configure WT for at startup in riak_kv_wterl_backend:max_sessions/1.
  • Make sure Erlang is optimizing for selective receive in async_nif_enqueue/3 because in the eLevelDB driver there is a comment: "This cannot be a separate function. Code must be inline to trigger Erlang compiler's use of optimized selective receive."
  • Provide a way to configure the cursor options, right now they are always "raw,overwrite".
  • Add support for Riak/KV 2i indexes using the same design pattern as eLevelDB (in a future version consider alternate schema)
  • If an operation using a shared cursor results in a non-normal error then it should be closed/discarded from the recycled pool
  • Cache cursors based on hash(table/config) rather than just table.
  • Finish NIF unload/reload functions and test.
  • Test an upgrade, include a format/schema/WT change.
  • When WT_PANIC is returned first try to unload/reload then driver and reset all state, if that fails then exit gracefully.
  • Currently the riak_kv_wterl_backend module is stored in this repository, but it really belongs in the riak_kv repository.
  • wterl:truncate/5 can segv, and its tests are commented out
  • Add async_nif and wterl NIF stats to the results provided by the stats API
  • Longer term ideas/changes to consider:
    • More testing, especially pulse/qc
    • Riak/KV integration
      • Store 2i indexes in separate tables
      • Store buckets, in separate tables and keep a <<bucket/key>> index to ensure that folds across a vnode are easy
      • Provide a drop bucket API call
      • Support key expirey
      • An ets API (like the LevelDB's lets project)
      • Use mime-type to inform WT's schema for key value encoding
  • Other use cases within Riak
    • An AAE driver using WT
    • An ability to store the ring file via WT

Deploying

You can deploy wterl into a Riak devrel cluster using the enable-wterl script. Clone the riak repo, change your working directory to it, and then execute the enable-wterl script. It adds wterl as a dependency, runs make all devrel, and then modifies the configuration settings of the resulting dev nodes to use the WiredTiger storage backend.