Commit graph

295 commits

Author SHA1 Message Date
Gregory Burd
13a9036764 So, WiredTiger prefers that extensions be shared libraries however those
extensions reference other shared libraries.  Previously I tried to turn
all those shared libs into static libs, however that didn't pan out so well.
So now the libwiredtiger.so library is loaded by wterl.so also, later on
during use libwiredtiger.so will load libwiredtiger_snappy.so and
libwiredtiger_bzip2.so which now have libsnappy.a and libbz2.a respectively
linked statically so as to avoid confusion.
2013-04-16 19:31:10 -04:00
Gregory Burd
76d7a7d0a3 Revert a small change 2013-04-16 17:20:57 -04:00
Gregory Burd
2d077a0baf Explicitly close cursors we open despite knowing they'll be closed with the session later. 2013-04-16 17:15:23 -04:00
Gregory Burd
3dab6a2dc5 Clean up 2013-04-16 17:09:34 -04:00
Gregory Burd
36c1d3f829 Move things around in the build script to get a bit of reuse. 2013-04-16 16:24:04 -04:00
Gregory Burd
bd4d852a19 Remove wildcards from the patch file names and fix checkout of wiredtiger repo to pull and track remote branch. 2013-04-16 17:41:00 +00:00
Gregory Burd
249c600554 Switching back to static linking to avoid platform issues and have
confidence when in deployment of what code is being used.
2013-04-16 11:47:04 -04:00
Gregory Burd
941bb0a929 Whitespace 2013-04-16 11:46:51 -04:00
Gregory Burd
ba41dd7fb6 Use the table name in get/put/delete calls to form an "affinity" with a
worker queue so that we spread work around and make it more likely that
work for a given table goes first to a given set of worker threads.
2013-04-15 18:46:06 -04:00
Gregory Burd
371779d14e Return to alloc'ed requests because there may be many more in flight
than those in the various queues.  Reenable the (still failing)
truncate tests (because they don't SEGV anymore).  Still might be
a memory leak, next up is valgrind.
2013-04-15 17:37:14 -04:00
Gregory Burd
668109de25 Added match/gt/lt atom return for cursor_search_near() call. Changed
the request queue over to a simple fifo queue which could (if needed)
be made lock-free.  Cursor searches can optionally now specifiy that
they are mid-scan so as not to have their cursor handles reset every
call.
2013-04-15 15:22:12 -04:00
Gregory Burd
e37c5b68d9 Type-o fixed. 2013-04-15 00:25:00 -04:00
Gregory Burd
cc73310cb3 Missed a change. 2013-04-15 00:11:48 -04:00
Gregory Burd
5ba491adfa We don't use the scheduler id from Erlang anymore in async_nif so
stop sending it over.  Allow the user to set a "type" of storage
in their config to either 'table' for btree or 'lsm' for a log
structured merge tree.  Various other cleanup.
2013-04-15 00:08:01 -04:00
Gregory Burd
9ed2137730 A bit of cleanup 2013-04-14 19:33:14 -04:00
Super-User
15a9a70c41 bzero isn't portable, use memset instead. Include bzip2 and snappy source for use by WT block compression
e
2013-04-14 21:23:57 +00:00
Gregory Burd
22643fc312 Change from waking up all threads to waking up one thread. 2013-04-14 13:54:45 -04:00
Gregory Burd
8fe9dc9bad This is a different approach from async2, instead of a single queue
and many workers this has a queue per scheduler and a few workers per
queue.
2013-04-14 08:44:54 -04:00
Gregory Burd
456129e7f3 Reusing closed cursors is bad for your health (SEGV), don't do that.
Also a bit of paranoia, bzero after all enif_alloc calls.
2013-04-12 16:59:10 -04:00
Gregory Burd
1962640382 Move the cleanup code into conn_close and remove the need for a dtor
function for resource GC/cleanup.  It's easier.
2013-04-12 16:09:00 -04:00
Gregory Burd
b0ca1e4098 WIP -- 41 tests pass, 3 fail = progress! Right now I have the connection
dtor commented out, otherwise it would SEGV on GC.  Some of the truncate
tests fail (race?) but don't SEGV, so that's not so bad.  Fixed numerous
issues and also removed a mutex and queue of idle worker threads because
it isn't used so why bother with it?
2013-04-12 15:25:56 -04:00
Gregory Burd
5a73264307 WIP -- Most tests passing, truncate and drop failing due to EBUSY,
also not sure why there is a segv when exiting but not in gdb.
Unlock mutex before returning error codes.  Alloc/free copies of some
strings allocated on the stack.
2013-04-11 11:57:41 -04:00
Gregory Burd
5c0295624d WIP -- most tests running, a few commented out, sometimes a segv on exit 2013-04-08 17:21:48 -04:00
Gregory Burd
07061ed6e8 WIP -- more tests passing, plenty left to fix 2013-04-07 22:16:44 -04:00
Gregory Burd
3b41805a71 WIP -- continue to fix tests in wterl 2013-04-07 10:07:19 -04:00
Gregory Burd
5ac006630e WIP -- tests failing -- WIP
Fixed numerous bugs.  Pushed config_to_bin down to just before
calling the _nif() function, everywhere else config is a proplist.
2013-04-07 09:21:47 -04:00
Gregory Burd
2675aa501a Opening a connection now involves passing in both a configuration
string for the connection and for any sessions.  Don't convert
configuration proplists into strings (Erlang binaries) until calling
into the NIF layer.
2013-04-06 17:19:59 -04:00
Gregory Burd
07592f20fb Use non-cursor ops for get/put/delete because under the covers they
are reusing sessions and cursors in the NIF layer.  Keep two cursors
open for just the house keeping tasks, is_empty and stats.
2013-04-06 17:17:50 -04:00
Gregory Burd
f7219dcaec WIP: Fix minor C89 issue 2013-04-06 11:14:23 -04:00
Gregory Burd
b4f82a388d WIP-- Compiling, not yet tested/functional -- WIP
Changes required to iron out compiler errors, warnings, etc.  Code now
compiles with clang or gcc.
2013-04-06 11:05:41 -04:00
Gregory Burd
19268b7c77 WIP-- Not Yet Compiling or Functional -- WIP
* No longer expose WT_SESSION into Erlang at all as WT's model is to
maintain one WT_SESSION per-thread and we don't know anything about
threads in Erlang.

* async_nif worker threads don't pull both mutexes on every loop
when processing requests, only one

* async_nif provides a worker_id (int, 0 - MAX_WORKERS) within the
work block scope which we use to find our per-worker WT_SESSIONs

* async_nif maintained a number of globals which I'm moving into
the NIF's priv_data so that on upgrade/reload we have a fighting
chance to "Do the Right Thing(TM)".

* NIF Upgrades/Reloads started to plumb this in.

* Use a khash to manage the cache of URI->WT_CURSORs per WT_SESSION.

* Added start/stop positions into truncate call to allow for truncating
sub-ranges data.

* Lots of other details I'm sure I've forgotten and more left undone.
Search for "TODO:" or try to compile to see what's left, and then
there is a need for a lot more tests given all this new complexity.
2013-04-05 18:09:54 -04:00
Gregory Burd
163a5073cb An updated Emacs cmode leads to all sorts of whitespace fixes intermixed with
some merge changes.
2013-04-02 09:42:07 -04:00
Gregory Burd
9c379a21dd Ignore a bit more. 2013-04-02 09:39:05 -04:00
Gregory Burd
254891e776 Bumping reductions manually will not be supported in future releases of the
BEAM and has dubious value anyway.  Remove that and ignore queue depth for
now.
2013-04-02 09:37:49 -04:00
Gregory Burd
609676917d Rename function to match new API. 2013-04-02 09:36:34 -04:00
Gregory Burd
6aa51437cc Merge remote-tracking branch 'origin/master' into gsb-async-nifs2
Conflicts:
	c_src/wterl.c
	src/riak_kv_wterl_backend.erl
	src/wterl.erl
2013-04-02 09:33:41 -04:00
Gregory Burd
a75d60bf27 Add an RPATH to the extionsion shared libs so they can find objects in the priv/ directory at runtime 2013-04-01 16:17:18 -04:00
Gregory Burd
95a6f52a1d Whitespace 2013-04-01 14:00:06 -04:00
Gregory Burd
e3219ff2c5 Add linker flags so that shared libraries are found relative to either the wterl layout or the Riak layout. 2013-03-28 22:21:36 -04:00
Gregory Burd
1595ce3096 Automatically load all WiredTiger extension shared libraries found in the priv dir at load time. 2013-03-28 10:45:24 -04:00
Gregory Burd
3ef48e0ac0 Change from static linking to dynamic. Enable snappy and bzip2 compression. 2013-03-28 10:24:26 -04:00
Gregory Burd
46214daa06 Remove check for other module. 2013-03-28 10:23:56 -04:00
Gregory Burd
77e0dacdb5 Use a set of {Session, Cursor} pairs ("passes") for calls into WiredTiger
to avoid a) opening a large number of Sessions or Cursors and b) avoid
reusing a Session/Cursor pair in more than one thread.  Each operation,
except folds, use the Session/Cursor pair corresponding to the scheduler
id (roughly "core") they are running on.  Each vnode instance will have
(count(schedulers) + count(active folds)) Session/Cursor pairs in use so
be sure to set session_max large enough when opening the connection to
WiredTiger.  This method will cause problems with scheduler threads going
to "sleep" so be sure to run schedmon to wake them up.
2013-03-26 14:17:21 -04:00
Gregory Burd
78e667bc09 Depend on async_nif.h 2013-03-26 13:02:45 -04:00
Gregory Burd
9f4e08ca6e Queue and execute work from scheduler threads on other threads to prevent schedulers from sleeping. 2013-03-24 21:00:48 -04:00
Gregory Burd
85b84a5343 Folds need a separate cursor, but not a separate session, so use
the shared connection and session.
2013-03-21 21:39:47 -04:00
Gregory Burd
9834f54991 Cover a few more corner cases when encoding cache size and other
values to config strings.
2013-03-21 21:05:01 -04:00
Gregory Burd
9302def7cc Lower cache size guess from 1/3 to 1/4 of available memory. Finally correct lsm_bloom_new/oldest config. 2013-03-21 18:56:01 -04:00
Gregory Burd
0203b06a61 Cursor delete operation only requires cursor handle and key (no need for a value) 2013-03-21 18:44:13 -04:00
Gregory Burd
a844dc5846 Whitespace 2013-03-21 14:37:37 -04:00