greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
243 commits 1 branch 0 tags 670 KiB
Commit graph

76 commits

Author SHA1 Message Date
sears
769b7615dd disable periodic force writes in merger (now handled by stasis instead); fix concurrency bug where merge held writelock while forcing data 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@3285 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-11-04 20:55:27 +00:00
sears
6f82ae268a remove more template stuff 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2668 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-06-09 01:16:55 +00:00
sears
ef4d360b68 memTreeComponent is no longer a template 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2667 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-06-09 00:27:57 +00:00
sears
ae7c827ab5 add hacky garbage collection mechanism 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2484 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-28 21:56:25 +00:00
sears
36790f6c2c add comment 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2455 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-24 21:09:54 +00:00
sears
5be84f59e8 add support for early tombstone dropping, extend datatuple format to support timestamps 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2451 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-22 22:54:49 +00:00
sears
5343f3021e fix broken merge logic. it was sometimes dropping live tuples, and never dropping tombstones... 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2449 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-22 17:40:37 +00:00
sears
0508f6cb40 initial wal implementation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2434 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-20 20:17:26 +00:00
sears
e5058da393 fix merge deadlock; port to fixed stasis buffer manager api; enable new buffer manager by default 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2330 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-03-25 20:05:49 +00:00
sears
dad55b564c Misc statistics cleanup; improved merge cost estimations during backpressure. 
Misc statistics cleanups include:
 - fixes for corner case where c0 is empty, and it would divide by zero
 - always compute number of tuples (in / out, but not base)
 - rework amount of memory allocated when --test and --newserver are passed in.



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1582 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-01-11 19:24:16 +00:00
sears
a6a0ce3f54 extended stats cleanup, verbose mode, documentation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1518 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-17 18:48:26 +00:00
sears
052508fb76 remove mergeStats->current_size field, which is derivable from other fields in mergeStats 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1508 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-15 00:15:59 +00:00
sears
cd27f50baf remove ltable->tsize and ltable->tree_bytes fields 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1507 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 23:06:21 +00:00
sears
73e72b47b2 more stats cleanups 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1493 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 01:49:23 +00:00
sears
43425128ba clean up initialization / marshalling code. Lets some of the public mergeStat fields to be protected 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1490 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 00:06:32 +00:00
sears
eb4446f409 readability fix: fields in mergeStats that are only used for computing extended stats for pretty print are now prefixed with "stats_", and can be optionally disabled at compile time 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1489 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-13 22:27:13 +00:00
sears
347a4126f3 remove the NO_SNOWSHOVEL c-preprocessor stuff. There is no way it was working anyway... 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1480 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 01:04:33 +00:00
sears
a00531ae6d simplified merge_scheduler 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 00:51:19 +00:00
sears
a8e905c0d6 fix deadlock on shutdown; be more stingy with bloom filter RAM with small indexes, add bloom filter to c2, default back to old buffer manager 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1468 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:49:13 +00:00
sears
ccf3895595 conservatively make the bloom filter really big (should behave well down to 100 byte keys, waste ram otherwise...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1238 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:55:29 +00:00
sears
f9a1f411e1 initial checkin of bloom filter logic for lsm trees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1233 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 23:05:45 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
960ff041e2 remove old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1055 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-24 00:40:48 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00
sears
b894cebaf7 add memTreeComponent::batchedRevalidatingIterator; amortize cost of red black latching during merge 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1017 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 17:29:25 +00:00
sears
dc75f6d1f0 Implemented snowshoveling, with caveats. 
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.

There are a number of important limitations:
 - statistics gathering is somewhat messed up
 - the c0-c1 merge thread polls for new data
 - on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge.  Instead, we should never finish a merge if c0 is less than N% full
 - this commit introduces yet another batch of nasty mutex contention issues



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:43:46 +00:00
sears
fded91fec6 two unrelated changes: release rb_mut before getting header_mut on read path (breaks delta tuples), and implement first-cut support for pulling in statistics when reopening logstores (required for correct recovery) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@868 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 20:03:05 +00:00
sears
8c35a1265a fix bug in overshoot_fudge computation; reduce number of knobs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@860 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 23:00:23 +00:00
sears
eda7c0fd12 enable force interval; play with some knobs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@854 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 02:07:55 +00:00
sears
2530009ec0 fix deadlock; do not clear need_tick bit until overshoot is less than zero; tweak constants accordingly (fewer mutex acquistions! less blockage!) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@852 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 00:06:46 +00:00
sears
0fb644640a improved concurrency for merger. not happy with worst case write latency on my dev machine anymore (seems to be less than 2.5 seconds, up from less than 1). 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@842 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-17 04:49:19 +00:00
sears
0ecd7491c7 fix minor stats bugs; pull most stats computation into update_progress. The idea is that update_progress and tick, which are easily amortized, are the only stats things that need to grab latches 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@827 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-05 00:41:52 +00:00
sears
eda514ddaa amortize the computation of statistics (but not mutex acuqisitions) across many tuple insertions. Also, refactor stats stuff with latch contention in mind (but it still grabs too many latches...). This commit causes check_tcpclient to emit warnings on my machine. Not sure if it will happen on production hardware 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@824 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-02 21:47:58 +00:00
sears
fcf0dcca99 cleanup / reimplement some of the statistics calculations. dial back periodically_force 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@821 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 21:20:26 +00:00
sears
4050475de9 more bugfixes / cleanup. latching is probably overly conservative at this point, but ycsb bulkload finally seems to work 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@819 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 01:29:10 +00:00
sears
d981d91dfa numerous bugfixes; rate limit progress meter display 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@816 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 23:15:24 +00:00
sears
6eecf1557a restore support for concurrent disk reads; logserver was accidentally using a findTuple function that assumed diffs. This commit uses rwlc, and requires Stasis -r1393 or better 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@814 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 01:49:27 +00:00
sears
aa7f7189d6 admission control *almost* works; c1 sometimes blocks for a long time when a new c2 is created 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@809 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-26 00:58:17 +00:00
sears
c50c14d259 admission control is now behaving properly (but there are still no concurrent writes, or bounds on how long admission control throttles a given writer) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@806 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-21 23:43:17 +00:00
sears
2a1157602a Rewrite merge thread synchronization code, update status messages, implement preliminary (and ineffective) admission control, and force write merged data every megabyte (so that prograess can be tracked by admission control. Also, refactor quite a few things. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@805 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-19 23:42:06 +00:00
sears
1adb980555 added throttling policy and better stats. the throttling makes a merge policy bug painfully obvious (committing because i plan to do some refactoring...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@802 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-12 22:16:41 +00:00
sears
7b2560d906 initial merge manager (multi-merge) statistics 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@797 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 23:13:04 +00:00
sears
ce48882946 oops; forgot to add mergeStats.h; finished encapsuating its state 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@795 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 01:03:56 +00:00
sears
176a1e804a more encapsulation changes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@794 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 00:57:48 +00:00
sears
ded9cc2585 fixed rounding errors in statistics written to stdout; whitespace cleanup; more encapsulation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@793 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 21:29:15 +00:00
sears
151373d09a refactoring; move init_stasis into logtable, put merge stats into their own module 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@792 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 19:21:25 +00:00
sears
2ea8b9ff44 latching fixes, server no longer ignores --test, performance tuning 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@759 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-24 20:30:35 +00:00
sears
e1c937a602 Refactoring of logtable. Cleaned a bunch of includes, and logtable is now a template. This is in preparation for the new merge policy 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@698 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-17 21:51:26 +00:00
sears
6cd1ccb5ff commented about merge transaction strategy 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@697 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-16 17:58:19 +00:00