greg/wterl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
224 commits 19 branches 0 tags 33 MiB
Commit graph

117 commits

Author SHA1 Message Date
Gregory Burd
2a4b8ee7d2 WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
Gregory Burd
0fef28de92 WIP: basho_bench tests are running fine now, need more work to ensure cache is functioning properly. 2013-06-05 11:41:41 -04:00
Gregory Burd
778ba20352 WIP: a bit of cleanup fixes a few mistakes 2013-06-04 17:21:50 -04:00
Gregory Burd
f1b7d8322d WIP: replcae the kbtree with khash, we don't need the tree features (yet, if ever) and hash is faster; add a most-recently-used stash for contexts as it's highly likely that worker threads will do many operations with the same shape/signature of session/cursors/tables/config and that path can be lock-free as well making it much faster (one would hope); somewhere something is stepping on read-only ErlNifBinary data and so a crc check is failing and causing the runtime to abort, that's the latest item to find/fix. 2013-06-04 14:45:23 -04:00
Gregory Burd
b002294c4e WIP: all tests (but drop) passing again, need to fix that and valgrind next. 2013-05-31 20:32:02 -04:00
Gregory Burd
9468870e1f WIP: use a log2 histogram to track how long items live in the cache, evict items which have been in cache greater than the mean time. 2013-05-30 17:10:51 -04:00
Gregory Burd
a2cd1d562c WIP: devising a better way to cache/reuse session/cursor pairs. 2013-05-28 16:14:19 -04:00
Gregory Burd
013251e6d9 Update WiredTiger statistics less frequently and fix a build script 
mistake preventing checking out by branch name.
 2013-05-14 13:51:44 -04:00
Gregory Burd
0bde789af5 Tune the LSM a bit based on testing, increase the chunk size and add more merge 
worker threads into the mix.
 2013-05-01 07:06:13 -04:00
Gregory Burd
f1cb6f8c61 Remove support for bzip2. Change build_deps.sh to only remove the 
build files from wiredtiger when cleaning rather than all the files.
 2013-04-26 10:34:57 -04:00
Gregory Burd
422dcfda89 Return 'eagain' when request queue is full and then try the request again. 
In the worst case is the request queue remains full and we loop between
the NIF and Erlang forever trying over and over to enqueue the request. If
that happens we shouldn't take schedulers offline as the NIF calls are fast
and we shouldn't run out of memory as that is bounded.  CPU will show a lot
of activity, but progress will continue in Erlang.
 2013-04-25 15:18:23 -04:00
Gregory Burd
35e36d1cae Whitespace 2013-04-24 14:28:24 -04:00
Gregory Burd
54c5158490 Merge pull request #7 from basho-labs/gsb-event-handlers 
Deliver WiredTiger event notifications (messages, errors, progress) into Erlang for logging
 2013-04-23 12:02:22 -07:00
Gregory Burd
35de23c0f3 Include the gen_server module that will log events using lager. 2013-04-23 08:22:48 -04:00
Gregory Burd
6f58faaba2 Turn on some of the verbose options, but leave most off. 2013-04-22 20:55:45 -04:00
Gregory Burd
fc59ae3f90 WIP: Finished transitioning to a gen_server and a few other tasks. This works 
except that when trying to enif_send a bunch of messages eventually one causes
a segv just after reporting: size_object: bad tag for 0x80
 2013-04-22 17:45:48 -04:00
Gregory Burd
652771003e WIP: a good start, I need to switch over wterl_event_handler to be a 
gen_server and I need to add a way to set the pid of the message handler
process to the NIF API.
 2013-04-22 09:52:21 -04:00
Gregory Burd
6c74c825e4 Avoid calling a process that may have already exited. 2013-04-20 13:30:09 -04:00
Gregory Burd
48141346f0 Remove redundant wt from filenames. 2013-04-20 12:49:14 -04:00
Gregory Burd
a1459ce607 It's statistics:<your table name here> not table:statistics (oops). 2013-04-20 12:33:36 -04:00
Gregory Burd
b69f35364b Use the less onerous version of statistics. 2013-04-20 12:27:42 -04:00
Gregory Burd
7380bd3bd7 Open the statistics table, not the actual table. 2013-04-20 12:08:55 -04:00
Gregory Burd
554aba4331 Truncate table isn't used by the WiredTiger Riak/KV backend API and it is 
currently failing.  Comment out of the tests, for now.
 2013-04-20 08:20:45 -04:00
Gregory Burd
be95229af9 On error when fetching stats, ignore and return whatever you have so far. 2013-04-19 16:44:51 -04:00
Gregory Burd
17cfdcd04f Fix default tuple. 2013-04-19 16:07:18 -04:00
Gregory Burd
e9145c344c Ignore eperm on status cursors and empty cursors (for now). 2013-04-19 15:46:58 -04:00
Gregory Burd
5d3cb9997a For now we ignore eperm and ebusy error returns on fold and drop. 2013-04-19 15:09:29 -04:00
Gregory Burd
d505f7f9c8 Whitespace. Call wterl:drop not truncate for drop calls and set force 
to true when calling so that we ignore ENOENT.  Change the session estimate
down a bit.
 2013-04-19 14:55:32 -04:00
Gregory Burd
95d8a28453 Remove unused num_contexts count. Cleanup the session/cursor cache 
init process.
 2013-04-18 13:36:24 -04:00
Gregory Burd
60dd048b7e Move the FIFO Queue implementation into its own file (fifo_q.h). Work 
on the nif_unload path.  Free up resources owned by wterl.c when
unloading.  Continue to evolve the build script.  Add to khash the ability
to create a hash that maps from a pointer to a value. There is still a segv
due to a race wterl.c:do_unload() which needs to be addressed.
 2013-04-18 10:37:36 -04:00
Gregory Burd
1ae8e5698f Ensure that the ratio of workers to queues is 2:1 and that there are at 
least 2 queues regardless.  Fix a few race conditions (h/t Sue from
WiredTiger for some nice work) and cherry pick (for now) a commit that
fixes a bug I triggered and Keith fixed (in < 10min from report) related
to WiredTiger stats.  Ensure that my guesstimate for session_max is no
larger than WiredTiger can manage.  Continue to fiddle with the build
script.
 2013-04-17 16:48:23 -04:00
Gregory Burd
f664c303d8 Disable prefix compression (for now). 2013-04-17 14:23:56 -04:00
Gregory Burd
123dfa600e Simplified the worker look function. Added ability to pick block 
compressor in config, default is snappy, off is {block_compressor, none}.
 2013-04-17 13:19:06 -04:00
Gregory Burd
87f70d75a1 Inline the fifo_q functions to speed them up and silence compiler warnings 
for unused API calls.  Add a fifo_q_full call to hide the details of that.
Alloc work queues along with the async_nif at the end of that memory block.
Fix a few places where things should be free'd and were not.  Change enqueue
to return 0 when shutting down.  Fix a race related to shutdown.  When I use
gdb eunit calls ?cmd() seem to fail, so I've created rmdir:path() to replace
?cmd("rm -rf path") calls.
 2013-04-17 11:17:13 -04:00
Gregory Burd
1913e7fdf5 Continue to iterate on the build system to accomodate shared libs. 2013-04-16 21:46:53 -04:00
Gregory Burd
36c1d3f829 Move things around in the build script to get a bit of reuse. 2013-04-16 16:24:04 -04:00
Gregory Burd
941bb0a929 Whitespace 2013-04-16 11:46:51 -04:00
Gregory Burd
371779d14e Return to alloc'ed requests because there may be many more in flight 
than those in the various queues.  Reenable the (still failing)
truncate tests (because they don't SEGV anymore).  Still might be
a memory leak, next up is valgrind.
 2013-04-15 17:37:14 -04:00
Gregory Burd
668109de25 Added match/gt/lt atom return for cursor_search_near() call. Changed 
the request queue over to a simple fifo queue which could (if needed)
be made lock-free.  Cursor searches can optionally now specifiy that
they are mid-scan so as not to have their cursor handles reset every
call.
 2013-04-15 15:22:12 -04:00
Gregory Burd
e37c5b68d9 Type-o fixed. 2013-04-15 00:25:00 -04:00
Gregory Burd
cc73310cb3 Missed a change. 2013-04-15 00:11:48 -04:00
Gregory Burd
5ba491adfa We don't use the scheduler id from Erlang anymore in async_nif so 
stop sending it over.  Allow the user to set a "type" of storage
in their config to either 'table' for btree or 'lsm' for a log
structured merge tree.  Various other cleanup.
 2013-04-15 00:08:01 -04:00
Gregory Burd
8fe9dc9bad This is a different approach from async2, instead of a single queue 
and many workers this has a queue per scheduler and a few workers per
queue.
 2013-04-14 08:44:54 -04:00
Gregory Burd
456129e7f3 Reusing closed cursors is bad for your health (SEGV), don't do that. 
Also a bit of paranoia, bzero after all enif_alloc calls.
 2013-04-12 16:59:10 -04:00
Gregory Burd
b0ca1e4098 WIP -- 41 tests pass, 3 fail = progress! Right now I have the connection 
dtor commented out, otherwise it would SEGV on GC.  Some of the truncate
tests fail (race?) but don't SEGV, so that's not so bad.  Fixed numerous
issues and also removed a mutex and queue of idle worker threads because
it isn't used so why bother with it?
 2013-04-12 15:25:56 -04:00
Gregory Burd
5a73264307 WIP -- Most tests passing, truncate and drop failing due to EBUSY, 
also not sure why there is a segv when exiting but not in gdb.
Unlock mutex before returning error codes.  Alloc/free copies of some
strings allocated on the stack.
 2013-04-11 11:57:41 -04:00
Gregory Burd
5c0295624d WIP -- most tests running, a few commented out, sometimes a segv on exit 2013-04-08 17:21:48 -04:00
Gregory Burd
07061ed6e8 WIP -- more tests passing, plenty left to fix 2013-04-07 22:16:44 -04:00
Gregory Burd
3b41805a71 WIP -- continue to fix tests in wterl 2013-04-07 10:07:19 -04:00
Gregory Burd
5ac006630e WIP -- tests failing -- WIP 
Fixed numerous bugs.  Pushed config_to_bin down to just before
calling the _nif() function, everywhere else config is a proplist.
 2013-04-07 09:21:47 -04:00