greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
242 commits 1 branch 0 tags 670 KiB
Commit graph

61 commits

Author SHA1 Message Date
surajach
5524b96bd9 Removing the typename keyword to make declarations compile with gcc 4.1.2 after a bunch of types are no longer templates 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2672 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-06-13 18:47:21 +00:00
sears
6f82ae268a remove more template stuff 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2668 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-06-09 01:16:55 +00:00
sears
ef4d360b68 memTreeComponent is no longer a template 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2667 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-06-09 00:27:57 +00:00
sears
72ecdea59e disable spammy printfs, change assert so that it prints extra debugging info, then ignores error 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2530 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-05-02 22:09:59 +00:00
sears
5be84f59e8 add support for early tombstone dropping, extend datatuple format to support timestamps 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2451 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-22 22:54:49 +00:00
sears
8135bbcc2e misc log cleanups; add command line parameter to newserver to control log mode 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2435 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-20 21:51:04 +00:00
sears
0508f6cb40 initial wal implementation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2434 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-20 20:17:26 +00:00
sears
e5058da393 fix merge deadlock; port to fixed stasis buffer manager api; enable new buffer manager by default 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2330 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-03-25 20:05:49 +00:00
sears
cd27f50baf remove ltable->tsize and ltable->tree_bytes fields 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1507 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 23:06:21 +00:00
sears
73e72b47b2 more stats cleanups 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1493 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 01:49:23 +00:00
sears
43425128ba clean up initialization / marshalling code. Lets some of the public mergeStat fields to be protected 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1490 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 00:06:32 +00:00
sears
a00531ae6d simplified merge_scheduler 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 00:51:19 +00:00
sears
a8e905c0d6 fix deadlock on shutdown; be more stingy with bloom filter RAM with small indexes, add bloom filter to c2, default back to old buffer manager 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1468 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:49:13 +00:00
sears
b9dd11a7e4 reduce default datapage_size to 1 (4096) to avoid useless seeks for small values 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1234 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:23:31 +00:00
sears
392a097e44 add atomic testAndSet operation (needed for sherpa tablet names' uniqid field) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1158 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 22:36:48 +00:00
sears
46eddd1269 amortize latches during scan 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1097 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:24 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00
sears
b894cebaf7 add memTreeComponent::batchedRevalidatingIterator; amortize cost of red black latching during merge 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1017 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 17:29:25 +00:00
sears
1e487bbc54 re-enable old merge policy for c1-c2 merger; implement shutdown (sort of) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1006 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-17 21:23:39 +00:00
sears
dc75f6d1f0 Implemented snowshoveling, with caveats. 
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.

There are a number of important limitations:
 - statistics gathering is somewhat messed up
 - the c0-c1 merge thread polls for new data
 - on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge.  Instead, we should never finish a merge if c0 is less than N% full
 - this commit introduces yet another batch of nasty mutex contention issues



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:43:46 +00:00
sears
cf6c6e5955 incease UPDATE_PROGRESS_DELTA by 10x, and FORCE_INTERVAL by 2.5x; tick no longer grabs a writelock on header_mut, and instead uses its own mutex 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@870 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 22:59:05 +00:00
sears
fded91fec6 two unrelated changes: release rb_mut before getting header_mut on read path (breaks delta tuples), and implement first-cut support for pulling in statistics when reopening logstores (required for correct recovery) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@868 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 20:03:05 +00:00
sears
4050475de9 more bugfixes / cleanup. latching is probably overly conservative at this point, but ycsb bulkload finally seems to work 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@819 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 01:29:10 +00:00
sears
d981d91dfa numerous bugfixes; rate limit progress meter display 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@816 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 23:15:24 +00:00
sears
6eecf1557a restore support for concurrent disk reads; logserver was accidentally using a findTuple function that assumed diffs. This commit uses rwlc, and requires Stasis -r1393 or better 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@814 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 01:49:27 +00:00
sears
aa7f7189d6 admission control *almost* works; c1 sometimes blocks for a long time when a new c2 is created 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@809 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-26 00:58:17 +00:00
sears
c50c14d259 admission control is now behaving properly (but there are still no concurrent writes, or bounds on how long admission control throttles a given writer) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@806 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-21 23:43:17 +00:00
sears
2a1157602a Rewrite merge thread synchronization code, update status messages, implement preliminary (and ineffective) admission control, and force write merged data every megabyte (so that prograess can be tracked by admission control. Also, refactor quite a few things. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@805 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-19 23:42:06 +00:00
sears
1adb980555 added throttling policy and better stats. the throttling makes a merge policy bug painfully obvious (committing because i plan to do some refactoring...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@802 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-12 22:16:41 +00:00
sears
7b2560d906 initial merge manager (multi-merge) statistics 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@797 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 23:13:04 +00:00
sears
151373d09a refactoring; move init_stasis into logtable, put merge stats into their own module 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@792 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 19:21:25 +00:00
sears
2ea8b9ff44 latching fixes, server no longer ignores --test, performance tuning 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@759 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-24 20:30:35 +00:00
sears
ead6207f86 add --test argument to server, fix segfault due to scans with empty c0_mergeable 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@699 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-17 23:45:41 +00:00
sears
e1c937a602 Refactoring of logtable. Cleaned a bunch of includes, and logtable is now a template. This is in preparation for the new merge policy 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@698 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-17 21:51:26 +00:00
sears
28433bf336 remove logiterators.h/cpp and stop #including datapage.cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@690 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-13 00:41:37 +00:00
sears
c8c48a188d more refactoring; all iterators now live in their respective tree components, fixed some hardcoded parameters; encapsulate allocators inside of diskTreeComponent 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@689 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-13 00:05:06 +00:00
sears
5f0d5c4f97 use RegionAllocator for InternalNodes and DataPages 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@685 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 23:17:03 +00:00
sears
54e73ab803 move the internal node region allocator stuff into diskTreeComponent::internalNode 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@681 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 20:23:15 +00:00
sears
01c65d7a3c diskTreeComponent -> diskTreeComponent::internalNodes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@680 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 19:47:12 +00:00
sears
1f2f4c745b more iterator renaming; also, getnext() -> next_callerFrees() 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@679 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 19:02:54 +00:00
sears
76cd6a26c7 changingMemTreeIterator -> memTreeComponent::changingMemTreeIterator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@676 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 17:47:47 +00:00
sears
de7984b05a create memTreeComponent class. Moved memTreeIterator, rbtree_t and tearDownTree into it (still need to move changableMemTreeIterator into it...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@673 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 01:42:23 +00:00
sears
747f4df1f3 Reopening existing trees now seems to work (no thorough tests yet). Added some network opcodes for statistics and debugging (not all are implemented yet). Reduce verbosity of server and client library. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@654 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-01 21:26:07 +00:00
sears
27720dddbe Implemented (but did not test) header update. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@650 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-27 00:35:13 +00:00
sears
51df61f8e9 logtableIterator now drops tombstone tuples by default 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@642 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-26 17:45:31 +00:00
sears
7c2397340c Fixed a bunch of iterator bugs and racy merges. Added drop_database utility. 
Bugfixes:
 - Atomically deallocate regions and update the logstore object
 - Proactively invalidate iterators after each merge (before, it would simply set a not-valid bit.  This doesn't work because iterators hold page pins, which breaks force-writes)
 - Clarify semantics of opening iterators mid-stream:  All calls now return iterators that return first key >= the requested one.  revalidate() needs the key > the requested one, so it calls peek(), then (if necessary) getnext().
 - Add asserts to check that the header is latched at update, and that tuples returned by iterators are strictly monotonically increasing'
 - Improve error handling in network.h  We still get (and terminate on) SIGPIPE.

Refactoring:
 - Add dispatch function to network.h.



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@620 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 01:29:32 +00:00
sears
63a14d8509 network iterator API is now working (though it could use a better test suite) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@601 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-20 01:18:39 +00:00
sears
57c9afc8d8 rename logtree to diskTreeComponent 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@591 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:59:14 +00:00
sears
2b82a62a34 broke logstore.cpp into two pieces 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@590 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:52:21 +00:00