Gregory Burd
2ddf0da53e
Use malloc/free rather than enif_alloc/enif_free so as to avoid BEAM allocator
...
overhead (bytes and time). Create static references to commonly used Erlang
atoms to avoid overhead re-creating them on each request cycle.
2013-08-21 12:20:19 -04:00
Gregory Burd
83c3faf74f
Use malloc/free rather than enif_alloc/enif_free so as to avoid BEAM allocator
...
overhead (bytes and time).
2013-08-21 12:18:24 -04:00
Gregory Burd
2043e8ccc6
Because the build decends into the ext/compressors/snappy directory the
...
relative paths won't find system/include, so use the absolute paths instead.
2013-08-21 12:17:18 -04:00
Gregory Burd
33c8e53ccf
Update to latest release of WiredTiger. Also, make sure Snappy builds before WiredTiger.
2013-08-21 12:16:24 -04:00
Gregory Burd
e67da86a9b
Change backpressure method from EAGAIN to bump_reductions so as not to block Riak/KV vnode processes when queues backup.
2013-08-19 13:32:58 -04:00
Gregory Burd
ee904b4769
Lower the queue size to shrink potential for latency in queue. Remove earlier idea that more queues would lead to more even worker progress, just have 1 queue per Erlang-scheduler thread (generally, 1 per CPU core available). Also change the way worker threads decide when to cond_wait or migrate to other queues looking for work.
2013-07-31 15:06:28 -04:00
Gregory Burd
c9a4ab8325
Revert changes to async_nif and re-enable stats. Fixed selective recv.
2013-07-31 09:41:36 -04:00
Gregory Burd
4418a74183
Increase the number of queues for work to reside. Worker threads, once started, don't exit until shutdown.
2013-07-30 14:21:26 -04:00
Gregory Burd
1623d5293c
Increase the max queue size.
2013-07-30 13:30:43 -04:00
Gregory Burd
cce163db9f
Fix potential to use uninitialized value when branching.
2013-07-26 20:08:49 -04:00
Gregory Burd
9a5defd8c9
Merge remote-tracking branch 'origin/master' into gsb-workers-migrate
...
Conflicts:
c_src/async_nif.h
2013-07-26 10:31:23 -04:00
Gregory Burd
452d7694a6
Added some sanity checking of key/value sizes. Check for EAGAIN/INVAL/NOMEM when starting worker threads. Switch back to the 1.6.3 release branch of WT.
2013-07-26 10:27:21 -04:00
Gregory Burd
3627ff8690
Ensure that on EAGAIN we continue to try to spawn a worker. When workers finish with a queue have them migrate to the other queues looking for work.
2013-07-25 13:29:16 -04:00
Gregory Burd
2a847b82d0
Forgot to remove this when I dumped the MRU.
2013-07-18 13:21:22 -04:00
Gregory Burd
2694cc1dba
Remove the MRU, it wasn't really effective in most cases anyway and complicated logic (and was buggy). For now the cache mutex will be hot, but eventually I hope to move the cache to a lock-free dequeue. khash.h and cas.h aren't used anymore, so they have been removed.
2013-07-18 13:14:54 -04:00
Gregory Burd
bbadc81d53
Queue depth and num workers can race, so make sure that we start at least one worker when there are none active for that queue.
2013-07-15 16:51:08 -04:00
Gregory Burd
bd0323af7a
Update to WiredTiger 1.6.3. Fix a condition where a mutex was unlocked twice on eagain when queues were all full.
2013-07-15 12:21:10 -04:00
Gregory Burd
bc0f5dbfc7
Evict older half of items in the cache by removing items from the end of the list, don't waste cycles computing timestamps.
2013-07-02 22:23:32 -04:00
Gregory Burd
b727538162
Fix shutdown
2013-07-02 22:07:34 -04:00
Gregory Burd
2672bab3ea
Stats overhead due to hitting the clock and pulling a mutex caused a massive slowdown so now work is assigned to a new queue only when the candidate queue is deeper than the average of the other queues and threads are created only when the depth of the queue is larger than the number of threads working on that queue.
2013-07-02 19:58:00 -04:00
Gregory Burd
00e5889ac9
Changed conditions for worker thread creation.
2013-07-02 16:46:04 -04:00
Gregory Burd
4300b3036f
Working on triggers that start/stop worker threads.
2013-07-01 21:09:21 -04:00
Gregory Burd
c7b45a7c2b
Still ironing out stats.
2013-06-27 10:57:41 -04:00
Gregory Burd
c41e411a92
Worker threads come and go as needed with a lower bound of 2 and an
...
upper bound of ASYNC_NIF_MAX_WORKERS. Stats were improved to use
thread local storage for measures. With stats working again wterl
uses them to determine who to evict. Wterl's signature calculation
for an operation wasn't correct and so the cache wasn't efficient at
all, this has been fixed.
2013-06-25 13:31:43 -04:00
Gregory Burd
a3c54b1610
Cleanup a bit.
2013-06-19 14:54:27 -04:00
Gregory Burd
0f180a6531
Fixed a few mistakes.
2013-06-19 14:37:30 -04:00
Gregory Burd
450299dc2d
Comment out a test that's not yet working.
2013-06-18 16:40:43 -04:00
Gregory Burd
4ae8ffb4cd
Update debugging messages a bit. Fix a bug in the signature function.
2013-06-18 13:49:25 -04:00
Gregory Burd
34e88c9234
Add some debugging output.
2013-06-18 13:12:10 -04:00
Gregory Burd
50e24d0f48
Add a longer, multi-table test. Use a release version of WiredTiger from now on.
2013-06-18 09:21:58 -04:00
Gregory Burd
53307e8c01
A great deal of cleanup. EUnit and EQC tests pass.
2013-06-14 16:57:53 -04:00
Gregory Burd
ff7d1d6e20
WIP: further simplifying context cache
2013-06-14 10:52:45 -04:00
Gregory Burd
7952358781
WIP: cache wasn't returning items found
2013-06-12 09:08:09 -04:00
Gregory Burd
4460434db1
WIP: remove potential for infinite loops with CAS and fix a few issues in async
2013-06-12 08:09:51 -04:00
Gregory Burd
110b482962
Some paranoia and a few fixes
2013-06-11 12:13:06 -04:00
Gregory Burd
8ea866bf20
Logic inversion on CAS() operation.
2013-06-10 16:18:55 -04:00
Gregory Burd
2a4b8ee7d2
WIP: simplify the cache from hash-of-lists to list; use a CAS() operation to protect the most-recently-used (mru) list.
2013-06-10 14:31:59 -04:00
b2c0b65114
Fixes for OS/X and whitespace cleanup.
2013-06-06 15:16:50 -04:00
Gregory Burd
0fef28de92
WIP: basho_bench tests are running fine now, need more work to ensure cache is functioning properly.
2013-06-05 11:41:41 -04:00
Gregory Burd
778ba20352
WIP: a bit of cleanup fixes a few mistakes
2013-06-04 17:21:50 -04:00
Gregory Burd
f1b7d8322d
WIP: replcae the kbtree with khash, we don't need the tree features (yet, if ever) and hash is faster; add a most-recently-used stash for contexts as it's highly likely that worker threads will do many operations with the same shape/signature of session/cursors/tables/config and that path can be lock-free as well making it much faster (one would hope); somewhere something is stepping on read-only ErlNifBinary data and so a crc check is failing and causing the runtime to abort, that's the latest item to find/fix.
2013-06-04 14:45:23 -04:00
Gregory Burd
b002294c4e
WIP: all tests (but drop) passing again, need to fix that and valgrind next.
2013-05-31 20:32:02 -04:00
Gregory Burd
9468870e1f
WIP: use a log2 histogram to track how long items live in the cache, evict items which have been in cache greater than the mean time.
2013-05-30 17:10:51 -04:00
Gregory Burd
15fbc71ea7
WIP: pieces in place, need to work out the kinks now.
2013-05-30 14:21:34 -04:00
Gregory Burd
f0d5baeb0e
WIP: more caching work, still not done.
2013-05-29 14:57:35 -04:00
Gregory Burd
a2cd1d562c
WIP: devising a better way to cache/reuse session/cursor pairs.
2013-05-28 16:14:19 -04:00
Gregory Burd
013251e6d9
Update WiredTiger statistics less frequently and fix a build script
...
mistake preventing checking out by branch name.
2013-05-14 13:51:44 -04:00
Gregory Burd
786142ce73
Add a bit of statistics tracking for two reasons, a) to help inform
...
where a request should be enqueded and b) to track request latency.
2013-05-01 22:02:37 -04:00
Gregory Burd
ae64a5e26f
Move async nif struct definition back to where it belongs.
2013-05-01 22:02:21 -04:00
Gregory Burd
d1d648af1e
Now matches against proper library name and prevents rebuilding when unnecessary.
2013-05-01 22:00:43 -04:00
