sears
07b35f4e20
scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-08-21 03:09:18 +00:00
sears
dc75f6d1f0
Implemented snowshoveling, with caveats.
...
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.
There are a number of important limitations:
- statistics gathering is somewhat messed up
- the c0-c1 merge thread polls for new data
- on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge. Instead, we should never finish a merge if c0 is less than N% full
- this commit introduces yet another batch of nasty mutex contention issues
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-08-05 17:43:46 +00:00
sears
9dcb624649
improved bps estimation; add "minisleeps" to update_progress; they do not belong there long term, but this is better than nothing
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@929 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-07-14 21:46:27 +00:00
sears
fded91fec6
two unrelated changes: release rb_mut before getting header_mut on read path (breaks delta tuples), and implement first-cut support for pulling in statistics when reopening logstores (required for correct recovery)
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@868 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-06-21 20:03:05 +00:00
sears
8c35a1265a
fix bug in overshoot_fudge computation; reduce number of knobs
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@860 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-06-18 23:00:23 +00:00
sears
0fb644640a
improved concurrency for merger. not happy with worst case write latency on my dev machine anymore (seems to be less than 2.5 seconds, up from less than 1).
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@842 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-06-17 04:49:19 +00:00
sears
0ecd7491c7
fix minor stats bugs; pull most stats computation into update_progress. The idea is that update_progress and tick, which are easily amortized, are the only stats things that need to grab latches
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@827 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-06-05 00:41:52 +00:00
sears
eda514ddaa
amortize the computation of statistics (but not mutex acuqisitions) across many tuple insertions. Also, refactor stats stuff with latch contention in mind (but it still grabs too many latches...). This commit causes check_tcpclient to emit warnings on my machine. Not sure if it will happen on production hardware
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@824 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-06-02 21:47:58 +00:00
sears
fcf0dcca99
cleanup / reimplement some of the statistics calculations. dial back periodically_force
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@821 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-05-28 21:20:26 +00:00
sears
aa7f7189d6
admission control *almost* works; c1 sometimes blocks for a long time when a new c2 is created
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@809 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-05-26 00:58:17 +00:00
sears
c50c14d259
admission control is now behaving properly (but there are still no concurrent writes, or bounds on how long admission control throttles a given writer)
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@806 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-05-21 23:43:17 +00:00
sears
2a1157602a
Rewrite merge thread synchronization code, update status messages, implement preliminary (and ineffective) admission control, and force write merged data every megabyte (so that prograess can be tracked by admission control. Also, refactor quite a few things.
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@805 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-05-19 23:42:06 +00:00
sears
1adb980555
added throttling policy and better stats. the throttling makes a merge policy bug painfully obvious (committing because i plan to do some refactoring...)
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@802 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-05-12 22:16:41 +00:00
sears
7b2560d906
initial merge manager (multi-merge) statistics
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@797 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-04-29 23:13:04 +00:00
sears
ce48882946
oops; forgot to add mergeStats.h; finished encapsuating its state
...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@795 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-04-29 01:03:56 +00:00
