greg/wterl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
269 commits 19 branches 0 tags 33 MiB
Commit graph

89 commits

Author SHA1 Message Date
Gregory Burd
9a5defd8c9 Merge remote-tracking branch 'origin/master' into gsb-workers-migrate 
Conflicts:
	c_src/async_nif.h
 2013-07-26 10:31:23 -04:00
Gregory Burd
452d7694a6 Added some sanity checking of key/value sizes. Check for EAGAIN/INVAL/NOMEM when starting worker threads. Switch back to the 1.6.3 release branch of WT. 2013-07-26 10:27:21 -04:00
Gregory Burd
3627ff8690 Ensure that on EAGAIN we continue to try to spawn a worker. When workers finish with a queue have them migrate to the other queues looking for work. 2013-07-25 13:29:16 -04:00
Gregory Burd
2a847b82d0 Forgot to remove this when I dumped the MRU. 2013-07-18 13:21:22 -04:00
Gregory Burd
2694cc1dba Remove the MRU, it wasn't really effective in most cases anyway and complicated logic (and was buggy). For now the cache mutex will be hot, but eventually I hope to move the cache to a lock-free dequeue. khash.h and cas.h aren't used anymore, so they have been removed. 2013-07-18 13:14:54 -04:00
Gregory Burd
bc0f5dbfc7 Evict older half of items in the cache by removing items from the end of the list, don't waste cycles computing timestamps. 2013-07-02 22:23:32 -04:00
Gregory Burd
00e5889ac9 Changed conditions for worker thread creation. 2013-07-02 16:46:04 -04:00
Gregory Burd
4300b3036f Working on triggers that start/stop worker threads. 2013-07-01 21:09:21 -04:00
Gregory Burd
c7b45a7c2b Still ironing out stats. 2013-06-27 10:57:41 -04:00
Gregory Burd
c41e411a92 Worker threads come and go as needed with a lower bound of 2 and an 
upper bound of ASYNC_NIF_MAX_WORKERS.  Stats were improved to use
thread local storage for measures.  With stats working again wterl
uses them to determine who to evict.  Wterl's signature calculation
for an operation wasn't correct and so the cache wasn't efficient at
all, this has been fixed.
 2013-06-25 13:31:43 -04:00
Gregory Burd
0f180a6531 Fixed a few mistakes. 2013-06-19 14:37:30 -04:00
Gregory Burd
450299dc2d Comment out a test that's not yet working. 2013-06-18 16:40:43 -04:00
Gregory Burd
4ae8ffb4cd Update debugging messages a bit. Fix a bug in the signature function. 2013-06-18 13:49:25 -04:00
Gregory Burd
50e24d0f48 Add a longer, multi-table test. Use a release version of WiredTiger from now on. 2013-06-18 09:21:58 -04:00
Gregory Burd
53307e8c01 A great deal of cleanup. EUnit and EQC tests pass. 2013-06-14 16:57:53 -04:00
Gregory Burd
ff7d1d6e20 WIP: further simplifying context cache 2013-06-14 10:52:45 -04:00
Gregory Burd
7952358781 WIP: cache wasn't returning items found 2013-06-12 09:08:09 -04:00
Gregory Burd
4460434db1 WIP: remove potential for infinite loops with CAS and fix a few issues in async 2013-06-12 08:09:51 -04:00
Gregory Burd
110b482962 Some paranoia and a few fixes 2013-06-11 12:13:06 -04:00
Gregory Burd
8ea866bf20 Logic inversion on CAS() operation. 2013-06-10 16:18:55 -04:00
Gregory Burd
2a4b8ee7d2 WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list. 2013-06-10 14:31:59 -04:00
Gregory Burd
b2c0b65114 Fixes for OS/X and whitespace cleanup. 2013-06-06 15:16:50 -04:00
Gregory Burd
778ba20352 WIP: a bit of cleanup fixes a few mistakes 2013-06-04 17:21:50 -04:00
Gregory Burd
f1b7d8322d WIP: replcae the kbtree with khash, we don't need the tree features (yet, if ever) and hash is faster; add a most-recently-used stash for contexts as it's highly likely that worker threads will do many operations with the same shape/signature of session/cursors/tables/config and that path can be lock-free as well making it much faster (one would hope); somewhere something is stepping on read-only ErlNifBinary data and so a crc check is failing and causing the runtime to abort, that's the latest item to find/fix. 2013-06-04 14:45:23 -04:00
Gregory Burd
b002294c4e WIP: all tests (but drop) passing again, need to fix that and valgrind next. 2013-05-31 20:32:02 -04:00
Gregory Burd
9468870e1f WIP: use a log2 histogram to track how long items live in the cache, evict items which have been in cache greater than the mean time. 2013-05-30 17:10:51 -04:00
Gregory Burd
15fbc71ea7 WIP: pieces in place, need to work out the kinks now. 2013-05-30 14:21:34 -04:00
Gregory Burd
f0d5baeb0e WIP: more caching work, still not done. 2013-05-29 14:57:35 -04:00
Gregory Burd
a2cd1d562c WIP: devising a better way to cache/reuse session/cursor pairs. 2013-05-28 16:14:19 -04:00
Gregory Burd
12fd9134fa Add a few placeholders for missing functions. 2013-05-01 21:59:12 -04:00
Gregory Burd
fc59ae3f90 WIP: Finished transitioning to a gen_server and a few other tasks. This works 
except that when trying to enif_send a bunch of messages eventually one causes
a segv just after reporting: size_object: bad tag for 0x80
 2013-04-22 17:45:48 -04:00
Gregory Burd
652771003e WIP: a good start, I need to switch over wterl_event_handler to be a 
gen_server and I need to add a way to set the pid of the message handler
process to the NIF API.
 2013-04-22 09:52:21 -04:00
Gregory Burd
ac59cd69dd Check return from kn_init. WiredTiger truncate doesn't require you to position cursors for the range to be truncated. 2013-04-21 11:14:48 -04:00
Gregory Burd
3310129918 * Check to make sure that all ASYNC_NIF_REPLY() calls are at the end of their 
blocks, just before return and after releasing any local resources or locks.

* Check the return value of when setting up cursor caches so as not to miss an
ENOMEM or other error.

*Cleanup and free resources when closing a connection handle.

* Add a few missing mutex unlock calls on error paths.

* Ensure all resources are free'd/released/closed during truncate error paths.

* Free up alloc'ed copies of keys, cursor handles and sessions on unload.
 2013-04-20 07:38:11 -04:00
Gregory Burd
01a79a08c0 Add a comment and remove an mutext unlock on error because this function is not 
where the lock is acquired.
 2013-04-20 06:13:41 -04:00
Gregory Burd
b936b99531 Minor fix. 2013-04-19 15:00:57 -04:00
Gregory Burd
d505f7f9c8 Whitespace. Call wterl:drop not truncate for drop calls and set force 
to true when calling so that we ignore ENOENT.  Change the session estimate
down a bit.
 2013-04-19 14:55:32 -04:00
Gregory Burd
8d8ceecc8b enif_get_string can return < 1 when it copies less than the buffer size 
you pass into it, that'd result in a non-zero (aka true) test when in fact
it's a problem if the argument isn't passed completely (however unlikely
that is).

enif_alloc_env() requires that later you enif_free_env() which I wasn't doing,
this seems to keep memory steady in test runs.
 2013-04-19 09:11:41 -04:00
Gregory Burd
95d8a28453 Remove unused num_contexts count. Cleanup the session/cursor cache 
init process.
 2013-04-18 13:36:24 -04:00
Gregory Burd
728d2281e0 Ensure that we init the shared cache when asking for a cursor before 
asking for a session.
 2013-04-18 12:24:49 -04:00
Gregory Burd
846f7f72ba Must compile before commit, must compile before commit... 2013-04-18 12:01:34 -04:00
Gregory Burd
7a148b9f19 get, put, delete don't use sessions, only cursors so don't fetch the 
shared session.
 2013-04-18 11:59:10 -04:00
Gregory Burd
83bca853f3 Store copies of Uri as keys in the hash table. 2013-04-18 11:47:12 -04:00
Gregory Burd
60dd048b7e Move the FIFO Queue implementation into its own file (fifo_q.h). Work 
on the nif_unload path.  Free up resources owned by wterl.c when
unloading.  Continue to evolve the build script.  Add to khash the ability
to create a hash that maps from a pointer to a value. There is still a segv
due to a race wterl.c:do_unload() which needs to be addressed.
 2013-04-18 10:37:36 -04:00
Gregory Burd
db953f5b39 Moved num_queue estimate earlier so as to ensure that the amount of 
memory allocated, zero'ed and free'd was consistent.  Skip free'ing
async environment as Erlang will free that for us when we no longer
reference it.  Fix the memory leak (or at least one of them) by no
longer copying the Uri into the hash table.
 2013-04-17 18:26:59 -04:00
Gregory Burd
87f70d75a1 Inline the fifo_q functions to speed them up and silence compiler warnings 
for unused API calls.  Add a fifo_q_full call to hide the details of that.
Alloc work queues along with the async_nif at the end of that memory block.
Fix a few places where things should be free'd and were not.  Change enqueue
to return 0 when shutting down.  Fix a race related to shutdown.  When I use
gdb eunit calls ?cmd() seem to fail, so I've created rmdir:path() to replace
?cmd("rm -rf path") calls.
 2013-04-17 11:17:13 -04:00
Gregory Burd
2d077a0baf Explicitly close cursors we open despite knowing they'll be closed with the session later. 2013-04-16 17:15:23 -04:00
Gregory Burd
ba41dd7fb6 Use the table name in get/put/delete calls to form an "affinity" with a 
worker queue so that we spread work around and make it more likely that
work for a given table goes first to a given set of worker threads.
 2013-04-15 18:46:06 -04:00
Gregory Burd
371779d14e Return to alloc'ed requests because there may be many more in flight 
than those in the various queues.  Reenable the (still failing)
truncate tests (because they don't SEGV anymore).  Still might be
a memory leak, next up is valgrind.
 2013-04-15 17:37:14 -04:00
Gregory Burd
668109de25 Added match/gt/lt atom return for cursor_search_near() call. Changed 
the request queue over to a simple fifo queue which could (if needed)
be made lock-free.  Cursor searches can optionally now specifiy that
they are mid-scan so as not to have their cursor handles reset every
call.
 2013-04-15 15:22:12 -04:00