f0d5baeb0eWIP: more caching work, still not done.
Gregory Burd
2013-05-29 14:56:38 -0400
a2cd1d562cWIP: devising a better way to cache/reuse session/cursor pairs.
Gregory Burd
2013-05-28 16:14:19 -0400
8e1e1b192dAllow clients of async_nif to set the number of concurrent workers.
gsb-affinity
Gregory Burd
2013-05-16 11:18:32 -0400
013251e6d9Update WiredTiger statistics less frequently and fix a build script mistake preventing checking out by branch name.
Gregory Burd
2013-05-14 13:51:44 -0400
4745950ad5Another TODO item.
Gregory Burd
2013-05-02 11:29:37 -0400
98fffd199fUpdated todo list.
Gregory Burd
2013-05-02 08:41:44 -0400
786142ce73Add a bit of statistics tracking for two reasons, a) to help inform where a request should be enqueded and b) to track request latency.
Gregory Burd
2013-05-01 22:02:37 -0400
ae64a5e26fMove async nif struct definition back to where it belongs.
Gregory Burd
2013-05-01 22:02:21 -0400
d1d648af1eNow matches against proper library name and prevents rebuilding when unnecessary.
Gregory Burd
2013-05-01 22:00:43 -0400
7c9c3b78dbRemove this vestige of bzip2 support.
Gregory Burd
2013-05-01 21:59:54 -0400
12fd9134faAdd a few placeholders for missing functions.
Gregory Burd
2013-05-01 21:59:12 -0400
0bde789af5Tune the LSM a bit based on testing, increase the chunk size and add more merge worker threads into the mix.
Gregory Burd
2013-05-01 07:06:13 -0400
d93f0bc162A more friendly approach to make clean in c_src.
Gregory Burd
2013-04-26 18:57:17 -0400
bfc180eeb3Merge branch 'master' of github.com:basho-labs/wterl
Gregory Burd
2013-04-26 10:37:06 -0400
f1cb6f8c61Remove support for bzip2. Change build_deps.sh to only remove the build files from wiredtiger when cleaning rather than all the files.
Gregory Burd
2013-04-26 10:34:57 -0400
eafee02865Only start 2 * num_queues worker threads initially. num_queues is generally equal to the number of cores reported by Erlang (info.scheduler_threads) which is either determined automatically by the Erlang BEAM runtime or via the +S flag. The minimum num_queues is 2, so the minimum number of workers is 4. The maximum number of workers is ASYNC_NIF_MAX_WORKER_QUEUE_SIZE (currently set to 128), but that would only happen if there were 64 cores (or you set +S 64:64 at startup).
Gregory Burd
2013-04-26 10:15:15 -0400
c410312c40Switch to the develop branch.
Gregory Burd
2013-04-26 10:07:10 -0400
3a51afcc20Copy paste error in patch file prevented proper build arguments for bzip2.
Gregory Burd
2013-04-25 16:36:17 -0400
422dcfda89Return 'eagain' when request queue is full and then try the request again. In the worst case is the request queue remains full and we loop between the NIF and Erlang forever trying over and over to enqueue the request. If that happens we shouldn't take schedulers offline as the NIF calls are fast and we shouldn't run out of memory as that is bounded. CPU will show a lot of activity, but progress will continue in Erlang.
Gregory Burd
2013-04-25 15:18:23 -0400
6b393ac47cKeep allocated req and ErlNifEnv around for reuse rather than re-alloc'ing them on each request should save us some overhead on the hot path.
gsb-recycle-reqs
Gregory Burd
2013-04-25 11:30:11 -0400
fc59ae3f90WIP: Finished transitioning to a gen_server and a few other tasks. This works except that when trying to enif_send a bunch of messages eventually one causes a segv just after reporting: size_object: bad tag for 0x80
Gregory Burd
2013-04-22 17:45:48 -0400
652771003eWIP: a good start, I need to switch over wterl_event_handler to be a gen_server and I need to add a way to set the pid of the message handler process to the NIF API.
Gregory Burd
2013-04-22 09:52:21 -0400
3398540831Have Erlang and C compilers be a bit more picky.
Gregory Burd
2013-04-21 11:16:04 -0400
ac59cd69ddCheck return from kn_init. WiredTiger truncate doesn't require you to position cursors for the range to be truncated.
Gregory Burd
2013-04-21 11:14:48 -0400
fae6831580Ensure that the env is clear when signaling shutdown.
Gregory Burd
2013-04-21 11:11:17 -0400
a282ad2f1dImprove, extend some targets.
Gregory Burd
2013-04-21 11:10:06 -0400
6c74c825e4Avoid calling a process that may have already exited.
gsb-async-nifs3
Gregory Burd
2013-04-20 13:30:09 -0400
48141346f0Remove redundant wt from filenames.
Gregory Burd
2013-04-20 12:49:14 -0400
a1459ce607It's statistics:<your table name here> not table:statistics (oops).
Gregory Burd
2013-04-20 12:33:36 -0400
b69f35364bUse the less onerous version of statistics.
Gregory Burd
2013-04-20 12:27:42 -0400
7380bd3bd7Open the statistics table, not the actual table.
Gregory Burd
2013-04-20 12:08:55 -0400
bfe56136d8* Be sure to release the reqs mutext on shutdown.
Gregory Burd
2013-04-20 08:28:38 -0400
554aba4331Truncate table isn't used by the WiredTiger Riak/KV backend API and it is currently failing. Comment out of the tests, for now.
Gregory Burd
2013-04-20 08:20:45 -0400
3310129918* Check to make sure that all ASYNC_NIF_REPLY() calls are at the end of their blocks, just before return and after releasing any local resources or locks.
Gregory Burd
2013-04-20 07:38:11 -0400
01a79a08c0Add a comment and remove an mutext unlock on error because this function is not where the lock is acquired.
Gregory Burd
2013-04-20 06:13:41 -0400
be95229af9On error when fetching stats, ignore and return whatever you have so far.
Gregory Burd
2013-04-19 16:44:51 -0400
d505f7f9c8Whitespace. Call wterl:drop not truncate for drop calls and set force to true when calling so that we ignore ENOENT. Change the session estimate down a bit.
Gregory Burd
2013-04-19 14:55:32 -0400
7b1dcb507dNeed to copy shared library links for bzip2 as well (duh).
Gregory Burd
2013-04-19 11:14:03 -0400
203fa1a54dMerge branch 'gsb-async-nifs3' of github.com:basho-labs/wterl into gsb-async-nifs3
Gregory Burd
2013-04-19 11:06:25 -0400
01a8926160Shared libs and files linking to them must be copied into our priv dir so that at runtime we can find the correct files.
Gregory Burd
2013-04-19 11:03:40 -0400
8d8ceecc8benif_get_string can return < 1 when it copies less than the buffer size you pass into it, that'd result in a non-zero (aka true) test when in fact it's a problem if the argument isn't passed completely (however unlikely that is).
Gregory Burd
2013-04-19 09:11:41 -0400
7a1e683a47Working toward a more automated valgrind.
Gregory Burd
2013-04-18 17:07:27 -0400
728d2281e0Ensure that we init the shared cache when asking for a cursor before asking for a session.
Gregory Burd
2013-04-18 12:24:49 -0400
846f7f72baMust compile before commit, must compile before commit...
Gregory Burd
2013-04-18 12:01:34 -0400
7a148b9f19get, put, delete don't use sessions, only cursors so don't fetch the shared session.
Gregory Burd
2013-04-18 11:59:10 -0400
83bca853f3Store copies of Uri as keys in the hash table.
Gregory Burd
2013-04-18 11:47:12 -0400
60dd048b7eMove the FIFO Queue implementation into its own file (fifo_q.h). Work on the nif_unload path. Free up resources owned by wterl.c when unloading. Continue to evolve the build script. Add to khash the ability to create a hash that maps from a pointer to a value. There is still a segv due to a race wterl.c:do_unload() which needs to be addressed.
Gregory Burd
2013-04-18 10:32:29 -0400
db953f5b39Moved num_queue estimate earlier so as to ensure that the amount of memory allocated, zero'ed and free'd was consistent. Skip free'ing async environment as Erlang will free that for us when we no longer reference it. Fix the memory leak (or at least one of them) by no longer copying the Uri into the hash table.
Gregory Burd
2013-04-17 18:26:59 -0400
1ae8e5698fEnsure that the ratio of workers to queues is 2:1 and that there are at least 2 queues regardless. Fix a few race conditions (h/t Sue from WiredTiger for some nice work) and cherry pick (for now) a commit that fixes a bug I triggered and Keith fixed (in < 10min from report) related to WiredTiger stats. Ensure that my guesstimate for session_max is no larger than WiredTiger can manage. Continue to fiddle with the build script.
Gregory Burd
2013-04-17 16:48:23 -0400
123dfa600eSimplified the worker look function. Added ability to pick block compressor in config, default is snappy, off is {block_compressor, none}.
Gregory Burd
2013-04-17 13:19:06 -0400
87f70d75a1Inline the fifo_q functions to speed them up and silence compiler warnings for unused API calls. Add a fifo_q_full call to hide the details of that. Alloc work queues along with the async_nif at the end of that memory block. Fix a few places where things should be free'd and were not. Change enqueue to return 0 when shutting down. Fix a race related to shutdown. When I use gdb eunit calls ?cmd() seem to fail, so I've created rmdir:path() to replace ?cmd("rm -rf path") calls.
Gregory Burd
2013-04-17 11:17:13 -0400
1913e7fdf5Continue to iterate on the build system to accomodate shared libs.
Gregory Burd
2013-04-16 21:46:53 -0400
13a9036764So, WiredTiger prefers that extensions be shared libraries however those extensions reference other shared libraries. Previously I tried to turn all those shared libs into static libs, however that didn't pan out so well. So now the libwiredtiger.so library is loaded by wterl.so also, later on during use libwiredtiger.so will load libwiredtiger_snappy.so and libwiredtiger_bzip2.so which now have libsnappy.a and libbz2.a respectively linked statically so as to avoid confusion.
Gregory Burd
2013-04-16 19:31:10 -0400
76d7a7d0a3Revert a small change
Gregory Burd
2013-04-16 17:20:57 -0400
2d077a0bafExplicitly close cursors we open despite knowing they'll be closed with the session later.
Gregory Burd
2013-04-16 17:15:23 -0400
3dab6a2dc5Clean up
Gregory Burd
2013-04-16 17:09:34 -0400
36c1d3f829Move things around in the build script to get a bit of reuse.
Gregory Burd
2013-04-16 16:24:04 -0400
bd4d852a19Remove wildcards from the patch file names and fix checkout of wiredtiger repo to pull and track remote branch.
Gregory Burd
2013-04-16 17:41:00 +0000
249c600554Switching back to static linking to avoid platform issues and have confidence when in deployment of what code is being used.
Gregory Burd
2013-04-16 11:47:04 -0400
ba41dd7fb6Use the table name in get/put/delete calls to form an "affinity" with a worker queue so that we spread work around and make it more likely that work for a given table goes first to a given set of worker threads.
Gregory Burd
2013-04-15 18:46:06 -0400
371779d14eReturn to alloc'ed requests because there may be many more in flight than those in the various queues. Reenable the (still failing) truncate tests (because they don't SEGV anymore). Still might be a memory leak, next up is valgrind.
Gregory Burd
2013-04-15 17:37:14 -0400
668109de25Added match/gt/lt atom return for cursor_search_near() call. Changed the request queue over to a simple fifo queue which could (if needed) be made lock-free. Cursor searches can optionally now specifiy that they are mid-scan so as not to have their cursor handles reset every call.
Gregory Burd
2013-04-15 15:22:12 -0400
cc73310cb3Missed a change.
Gregory Burd
2013-04-15 00:11:48 -0400
5ba491adfaWe don't use the scheduler id from Erlang anymore in async_nif so stop sending it over. Allow the user to set a "type" of storage in their config to either 'table' for btree or 'lsm' for a log structured merge tree. Various other cleanup.
Gregory Burd
2013-04-15 00:08:01 -0400
9ed2137730A bit of cleanup
Gregory Burd
2013-04-14 19:33:14 -0400
15a9a70c41bzero isn't portable, use memset instead. Include bzip2 and snappy source for use by WT block compression
Super-User
2013-04-14 21:23:57 +0000
22643fc312Change from waking up all threads to waking up one thread.
Gregory Burd
2013-04-14 13:54:45 -0400
8fe9dc9badThis is a different approach from async2, instead of a single queue and many workers this has a queue per scheduler and a few workers per queue.
Gregory Burd
2013-04-14 08:44:54 -0400
456129e7f3Reusing closed cursors is bad for your health (SEGV), don't do that. Also a bit of paranoia, bzero after all enif_alloc calls.
Gregory Burd
2013-04-12 16:59:10 -0400
1962640382Move the cleanup code into conn_close and remove the need for a dtor function for resource GC/cleanup. It's easier.
Gregory Burd
2013-04-12 16:09:00 -0400
b0ca1e4098WIP -- 41 tests pass, 3 fail = progress! Right now I have the connection dtor commented out, otherwise it would SEGV on GC. Some of the truncate tests fail (race?) but don't SEGV, so that's not so bad. Fixed numerous issues and also removed a mutex and queue of idle worker threads because it isn't used so why bother with it?
Gregory Burd
2013-04-12 15:25:56 -0400
5a73264307WIP -- Most tests passing, truncate and drop failing due to EBUSY, also not sure why there is a segv when exiting but not in gdb. Unlock mutex before returning error codes. Alloc/free copies of some strings allocated on the stack.
Gregory Burd
2013-04-11 11:57:41 -0400
5c0295624dWIP -- most tests running, a few commented out, sometimes a segv on exit
Gregory Burd
2013-04-08 17:21:48 -0400
07061ed6e8WIP -- more tests passing, plenty left to fix
Gregory Burd
2013-04-07 22:16:44 -0400
3b41805a71WIP -- continue to fix tests in wterl
Gregory Burd
2013-04-07 10:07:19 -0400
5ac006630eWIP -- tests failing -- WIP Fixed numerous bugs. Pushed config_to_bin down to just before calling the _nif() function, everywhere else config is a proplist.
Gregory Burd
2013-04-07 09:21:47 -0400
2675aa501aOpening a connection now involves passing in both a configuration string for the connection and for any sessions. Don't convert configuration proplists into strings (Erlang binaries) until calling into the NIF layer.
Gregory Burd
2013-04-06 17:19:59 -0400
07592f20fbUse non-cursor ops for get/put/delete because under the covers they are reusing sessions and cursors in the NIF layer. Keep two cursors open for just the house keeping tasks, is_empty and stats.
Gregory Burd
2013-04-06 17:17:50 -0400
Gregory Burd