on the nif_unload path. Free up resources owned by wterl.c when
unloading. Continue to evolve the build script. Add to khash the ability
to create a hash that maps from a pointer to a value. There is still a segv
due to a race wterl.c:do_unload() which needs to be addressed.
least 2 queues regardless. Fix a few race conditions (h/t Sue from
WiredTiger for some nice work) and cherry pick (for now) a commit that
fixes a bug I triggered and Keith fixed (in < 10min from report) related
to WiredTiger stats. Ensure that my guesstimate for session_max is no
larger than WiredTiger can manage. Continue to fiddle with the build
script.
for unused API calls. Add a fifo_q_full call to hide the details of that.
Alloc work queues along with the async_nif at the end of that memory block.
Fix a few places where things should be free'd and were not. Change enqueue
to return 0 when shutting down. Fix a race related to shutdown. When I use
gdb eunit calls ?cmd() seem to fail, so I've created rmdir:path() to replace
?cmd("rm -rf path") calls.
than those in the various queues. Reenable the (still failing)
truncate tests (because they don't SEGV anymore). Still might be
a memory leak, next up is valgrind.
the request queue over to a simple fifo queue which could (if needed)
be made lock-free. Cursor searches can optionally now specifiy that
they are mid-scan so as not to have their cursor handles reset every
call.
stop sending it over. Allow the user to set a "type" of storage
in their config to either 'table' for btree or 'lsm' for a log
structured merge tree. Various other cleanup.
dtor commented out, otherwise it would SEGV on GC. Some of the truncate
tests fail (race?) but don't SEGV, so that's not so bad. Fixed numerous
issues and also removed a mutex and queue of idle worker threads because
it isn't used so why bother with it?
also not sure why there is a segv when exiting but not in gdb.
Unlock mutex before returning error codes. Alloc/free copies of some
strings allocated on the stack.
* No longer expose WT_SESSION into Erlang at all as WT's model is to
maintain one WT_SESSION per-thread and we don't know anything about
threads in Erlang.
* async_nif worker threads don't pull both mutexes on every loop
when processing requests, only one
* async_nif provides a worker_id (int, 0 - MAX_WORKERS) within the
work block scope which we use to find our per-worker WT_SESSIONs
* async_nif maintained a number of globals which I'm moving into
the NIF's priv_data so that on upgrade/reload we have a fighting
chance to "Do the Right Thing(TM)".
* NIF Upgrades/Reloads started to plumb this in.
* Use a khash to manage the cache of URI->WT_CURSORs per WT_SESSION.
* Added start/stop positions into truncate call to allow for truncating
sub-ranges data.
* Lots of other details I'm sure I've forgotten and more left undone.
Search for "TODO:" or try to compile to see what's left, and then
there is a need for a lot more tests given all this new complexity.
to avoid a) opening a large number of Sessions or Cursors and b) avoid
reusing a Session/Cursor pair in more than one thread. Each operation,
except folds, use the Session/Cursor pair corresponding to the scheduler
id (roughly "core") they are running on. Each vnode instance will have
(count(schedulers) + count(active folds)) Session/Cursor pairs in use so
be sure to set session_max large enough when opening the connection to
WiredTiger. This method will cause problems with scheduler threads going
to "sleep" so be sure to run schedmon to wake them up.
reasonable setting for the WiredTiger cache size at runtime. This cache
is shared across all vnodes regarless of how many are active at any
given time. The algorithm is: max(1GB, 1/3 (RAM - Beam RSS size)). We don't
enable direct_io on purpose and data will be double buffered in WiredTiger's
cache and the filesystem buffer cache. This turns out to be faster than
direct I/O despite wasting a bit of RAM.
answer is 'no'). So far there has been no effort to validate that these
settings are in fact the best for Riak/KV or CS data access patterns. These
particular settings are, at best, an educated guess based on past experience,
the docs and reading about the benchmark the WiredTiger team published here:
https://github.com/wiredtiger/wiredtiger/wiki/YCSB-Mapkeeper-benchmark
Versions of WiredTiger prior to 1.3.3 had trouble handling riak restarts,
returning errors from wterl:session_create that would prevent riak from
restarting if it was not cleanly shut down on the previous run. We
previously avoided checking the return value of wterl:session_create to
avoid this problem. WT 1.3.3 fixes this, so reinstate the check of the
wterl:session_create return value.
Since 1.3.0 folded the session sync function into the session checkpoint
function, the wterl:sync/(2,3) functions are now renamed
wterl:checkpoint/(1,2). See
http://source.wiredtiger.com/1.3.0/upgrading.html for details.
Make the unit tests call riak_kv_backend:standard_test like other
backends do for their unit tests. This is a little awkward at the
moment because riak_kv_backend.erl doesn't live in this repository, so
it requires temporarily copying it into src to run "rebar eunit".
The backend fold_objects was failing because it was trying to use
wterl:fold_keys/3, which doesn't fold over values. Add fold/3 to
wterl.erl to fix this.
Also clean up a few dialyzer warnings.
WiredTiger session count defaults to 50, but we need at least one per
vnode, so for now hard-code it to 100. Also ignore table creation
results in the kv backend because WiredTiger is currently producing a
strange error if the table already exists.
In order to reduce the number of wiredtiger session open/close calls,
open a session at startup and keep it in the kv backend state. This is
safe as far as using these sessions on scheduler threads goes because
the riak_kv_vnode fsm serializes all calls through the backend, so
we'll never have a case of multiple scheduler threads concurrently
trying to use the same session.
Add a temporary copy of riak_kv_backend, renamed to
temp_riak_kv_backend, to allow wterl to build on its own. This file
must be deleted when riak_kv_wterl_backend is moved to its proper
place in riak_kv.
Change cursor_{next,prev} to return {ok, Key, Value}. Add
cursor_{next,prev}_{key,value} functions to allow cursor movement
followed by fetch of either key or value. Add fold_keys function. Add
unit tests for these changes. Also specify "inorder" on the unit test
generator functions, since the tests they specified are intended to be
run that way.
Break the single wterl unit test into a series of tests separated by
function. Use eunit assert macros and add test setup and teardown
functions. Add descriptive messages for the tests for display under
"rebar -v eunit".
Add app, supervisor, and worker to open and cache the database
connection. Revert to using binaries for WT config in order to allow
wterl callers to use proplists for configuration. Change {error,
not_found} to not_found return values from wterl.
Add support for cursor.{next,prev,search,search_near,reset}.
Change table_create to no longer pass back the table name, WiredTiger's
model is that you pass in the object name each time (and the current
model means we can't drop the table before we run the tests).
Upgrade to the WiredTiger 1.0 release, there's a bug where static
libraries aren't created correctly, for now create shared libraries
for wterl.
not a pure integer).
Remove "exclusive" from the list of options, it's not in the 1.0 release.
Set the cache_size for the test, just to prove it works.
A number of changes:
* Added exported functions to close connections and sessions.
* Added exported functions to create and drop tables. Currently not
happy with the shape of these functions, though, as they treat
tables in Erlang just as strings, and they force users of sessions
to have to pass both sessions and tables into get/put/delete
operations. It might be better to take a table identifier as part of
the session creation function and store that identifier with the
session resource.
* Added implementations for key-value get, put, and delete.
* Added very minimal unit tests for the changes above.
