Commit graph

116 commits

Author SHA1 Message Date
Gregory Burd
8e1e1b192d Allow clients of async_nif to set the number of concurrent workers. 2013-05-16 11:18:32 -04:00
Gregory Burd
013251e6d9 Update WiredTiger statistics less frequently and fix a build script
mistake preventing checking out by branch name.
2013-05-14 13:51:44 -04:00
Gregory Burd
786142ce73 Add a bit of statistics tracking for two reasons, a) to help inform
where a request should be enqueded and b) to track request latency.
2013-05-01 22:02:37 -04:00
Gregory Burd
ae64a5e26f Move async nif struct definition back to where it belongs. 2013-05-01 22:02:21 -04:00
Gregory Burd
d1d648af1e Now matches against proper library name and prevents rebuilding when unnecessary. 2013-05-01 22:00:43 -04:00
Gregory Burd
7c9c3b78db Remove this vestige of bzip2 support. 2013-05-01 21:59:54 -04:00
Gregory Burd
12fd9134fa Add a few placeholders for missing functions. 2013-05-01 21:59:12 -04:00
Gregory Burd
d93f0bc162 A more friendly approach to make clean in c_src. 2013-04-26 18:57:17 -04:00
Gregory Burd
bfc180eeb3 Merge branch 'master' of github.com:basho-labs/wterl
Conflicts:
	c_src/wiredtiger-build.patch
2013-04-26 10:37:06 -04:00
Gregory Burd
f1cb6f8c61 Remove support for bzip2. Change build_deps.sh to only remove the
build files from wiredtiger when cleaning rather than all the files.
2013-04-26 10:34:57 -04:00
Gregory Burd
eafee02865 Only start 2 * num_queues worker threads initially. num_queues is generally
equal to the number of cores reported by Erlang (info.scheduler_threads) which
is either determined automatically by the Erlang BEAM runtime or via the +S
flag.  The minimum num_queues is 2, so the minimum number of workers is 4.  The
maximum number of workers is ASYNC_NIF_MAX_WORKER_QUEUE_SIZE (currently set to
128), but that would only happen if there were 64 cores (or you set +S 64:64 at
startup).
2013-04-26 10:15:15 -04:00
Gregory Burd
c410312c40 Switch to the develop branch. 2013-04-26 10:07:10 -04:00
Gregory Burd
3a51afcc20 Copy paste error in patch file prevented proper build arguments for bzip2. 2013-04-25 16:36:17 -04:00
Gregory Burd
422dcfda89 Return 'eagain' when request queue is full and then try the request again.
In the worst case is the request queue remains full and we loop between
the NIF and Erlang forever trying over and over to enqueue the request. If
that happens we shouldn't take schedulers offline as the NIF calls are fast
and we shouldn't run out of memory as that is bounded.  CPU will show a lot
of activity, but progress will continue in Erlang.
2013-04-25 15:18:23 -04:00
Gregory Burd
6b393ac47c Keep allocated req and ErlNifEnv around for reuse rather than re-alloc'ing them on each request should save us some overhead on the hot path. 2013-04-25 11:30:11 -04:00
Gregory Burd
46243ff54a Remove debug compiler flags. 2013-04-23 15:00:44 -04:00
Gregory Burd
fc59ae3f90 WIP: Finished transitioning to a gen_server and a few other tasks. This works
except that when trying to enif_send a bunch of messages eventually one causes
a segv just after reporting: size_object: bad tag for 0x80
2013-04-22 17:45:48 -04:00
Gregory Burd
652771003e WIP: a good start, I need to switch over wterl_event_handler to be a
gen_server and I need to add a way to set the pid of the message handler
process to the NIF API.
2013-04-22 09:52:21 -04:00
Gregory Burd
ac59cd69dd Check return from kn_init. WiredTiger truncate doesn't require you to position cursors for the range to be truncated. 2013-04-21 11:14:48 -04:00
Gregory Burd
fae6831580 Ensure that the env is clear when signaling shutdown. 2013-04-21 11:11:17 -04:00
Gregory Burd
bfe56136d8 * Be sure to release the reqs mutext on shutdown. 2013-04-20 08:28:38 -04:00
Gregory Burd
3310129918 * Check to make sure that all ASYNC_NIF_REPLY() calls are at the end of their
blocks, just before return and after releasing any local resources or locks.

* Check the return value of when setting up cursor caches so as not to miss an
ENOMEM or other error.

*Cleanup and free resources when closing a connection handle.

* Add a few missing mutex unlock calls on error paths.

* Ensure all resources are free'd/released/closed during truncate error paths.

* Free up alloc'ed copies of keys, cursor handles and sessions on unload.
2013-04-20 07:38:11 -04:00
Gregory Burd
01a79a08c0 Add a comment and remove an mutext unlock on error because this function is not
where the lock is acquired.
2013-04-20 06:13:41 -04:00
Gregory Burd
b936b99531 Minor fix. 2013-04-19 15:00:57 -04:00
Gregory Burd
d505f7f9c8 Whitespace. Call wterl:drop not truncate for drop calls and set force
to true when calling so that we ignore ENOENT.  Change the session estimate
down a bit.
2013-04-19 14:55:32 -04:00
Gregory Burd
40bdda15bb Create links for bzip2 shared libraries. 2013-04-19 12:33:20 -04:00
Gregory Burd
7b1dcb507d Need to copy shared library links for bzip2 as well (duh). 2013-04-19 11:14:03 -04:00
Gregory Burd
01a8926160 Shared libs and files linking to them must be copied into our priv
dir so that at runtime we can find the correct files.
2013-04-19 11:03:40 -04:00
Gregory Burd
8d8ceecc8b enif_get_string can return < 1 when it copies less than the buffer size
you pass into it, that'd result in a non-zero (aka true) test when in fact
it's a problem if the argument isn't passed completely (however unlikely
that is).

enif_alloc_env() requires that later you enif_free_env() which I wasn't doing,
this seems to keep memory steady in test runs.
2013-04-19 09:11:41 -04:00
Gregory Burd
95d8a28453 Remove unused num_contexts count. Cleanup the session/cursor cache
init process.
2013-04-18 13:36:24 -04:00
Gregory Burd
728d2281e0 Ensure that we init the shared cache when asking for a cursor before
asking for a session.
2013-04-18 12:24:49 -04:00
Gregory Burd
846f7f72ba Must compile before commit, must compile before commit... 2013-04-18 12:01:34 -04:00
Gregory Burd
7a148b9f19 get, put, delete don't use sessions, only cursors so don't fetch the
shared session.
2013-04-18 11:59:10 -04:00
Gregory Burd
83bca853f3 Store copies of Uri as keys in the hash table. 2013-04-18 11:47:12 -04:00
Gregory Burd
60dd048b7e Move the FIFO Queue implementation into its own file (fifo_q.h). Work
on the nif_unload path.  Free up resources owned by wterl.c when
unloading.  Continue to evolve the build script.  Add to khash the ability
to create a hash that maps from a pointer to a value. There is still a segv
due to a race wterl.c:do_unload() which needs to be addressed.
2013-04-18 10:37:36 -04:00
Gregory Burd
db953f5b39 Moved num_queue estimate earlier so as to ensure that the amount of
memory allocated, zero'ed and free'd was consistent.  Skip free'ing
async environment as Erlang will free that for us when we no longer
reference it.  Fix the memory leak (or at least one of them) by no
longer copying the Uri into the hash table.
2013-04-17 18:26:59 -04:00
Gregory Burd
1ae8e5698f Ensure that the ratio of workers to queues is 2:1 and that there are at
least 2 queues regardless.  Fix a few race conditions (h/t Sue from
WiredTiger for some nice work) and cherry pick (for now) a commit that
fixes a bug I triggered and Keith fixed (in < 10min from report) related
to WiredTiger stats.  Ensure that my guesstimate for session_max is no
larger than WiredTiger can manage.  Continue to fiddle with the build
script.
2013-04-17 16:48:23 -04:00
Gregory Burd
123dfa600e Simplified the worker look function. Added ability to pick block
compressor in config, default is snappy, off is {block_compressor, none}.
2013-04-17 13:19:06 -04:00
Gregory Burd
87f70d75a1 Inline the fifo_q functions to speed them up and silence compiler warnings
for unused API calls.  Add a fifo_q_full call to hide the details of that.
Alloc work queues along with the async_nif at the end of that memory block.
Fix a few places where things should be free'd and were not.  Change enqueue
to return 0 when shutting down.  Fix a race related to shutdown.  When I use
gdb eunit calls ?cmd() seem to fail, so I've created rmdir:path() to replace
?cmd("rm -rf path") calls.
2013-04-17 11:17:13 -04:00
Gregory Burd
1913e7fdf5 Continue to iterate on the build system to accomodate shared libs. 2013-04-16 21:46:53 -04:00
Gregory Burd
5e80477d4a Merge remote-tracking branch 'origin/gsb-async-nifs3' into gsb-async-nifs3 2013-04-16 19:34:11 -04:00
Gregory Burd
13a9036764 So, WiredTiger prefers that extensions be shared libraries however those
extensions reference other shared libraries.  Previously I tried to turn
all those shared libs into static libs, however that didn't pan out so well.
So now the libwiredtiger.so library is loaded by wterl.so also, later on
during use libwiredtiger.so will load libwiredtiger_snappy.so and
libwiredtiger_bzip2.so which now have libsnappy.a and libbz2.a respectively
linked statically so as to avoid confusion.
2013-04-16 19:31:10 -04:00
Gregory Burd
2d077a0baf Explicitly close cursors we open despite knowing they'll be closed with the session later. 2013-04-16 17:15:23 -04:00
Gregory Burd
3dab6a2dc5 Clean up 2013-04-16 17:09:34 -04:00
Gregory Burd
36c1d3f829 Move things around in the build script to get a bit of reuse. 2013-04-16 16:24:04 -04:00
Gregory Burd
bd4d852a19 Remove wildcards from the patch file names and fix checkout of wiredtiger repo to pull and track remote branch. 2013-04-16 17:41:00 +00:00
Gregory Burd
249c600554 Switching back to static linking to avoid platform issues and have
confidence when in deployment of what code is being used.
2013-04-16 11:47:04 -04:00
Gregory Burd
ba41dd7fb6 Use the table name in get/put/delete calls to form an "affinity" with a
worker queue so that we spread work around and make it more likely that
work for a given table goes first to a given set of worker threads.
2013-04-15 18:46:06 -04:00
Gregory Burd
371779d14e Return to alloc'ed requests because there may be many more in flight
than those in the various queues.  Reenable the (still failing)
truncate tests (because they don't SEGV anymore).  Still might be
a memory leak, next up is valgrind.
2013-04-15 17:37:14 -04:00
Gregory Burd
668109de25 Added match/gt/lt atom return for cursor_search_near() call. Changed
the request queue over to a simple fifo queue which could (if needed)
be made lock-free.  Cursor searches can optionally now specifiy that
they are mid-scan so as not to have their cursor handles reset every
call.
2013-04-15 15:22:12 -04:00