greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
193 commits 1 branch 0 tags 670 KiB
Commit graph

58 commits

Author SHA1 Message Date
sears
8135bbcc2e misc log cleanups; add command line parameter to newserver to control log mode 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2435 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-04-20 21:51:04 +00:00
sears
83fe4b7703 fix array-out-of-bounds bug in unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1521 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-17 21:15:26 +00:00
sears
43425128ba clean up initialization / marshalling code. Lets some of the public mergeStat fields to be protected 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1490 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 00:06:32 +00:00
sears
a00531ae6d simplified merge_scheduler 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 00:51:19 +00:00
sears
8dddc7e168 remove line from unit test that had no effect, but broke compilation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1478 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-10 20:42:57 +00:00
sears
fbdc900623 fix #include mistake 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1239 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 18:27:56 +00:00
sears
8936df750a fix includes to avoid linker errors 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1236 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:52:32 +00:00
sears
3619e86271 initial checkin of bloom filter 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1232 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 18:06:19 +00:00
sears
6eaf5da8de port to newer gcc/g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1231 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-02 12:52:41 +00:00
sears
392a097e44 add atomic testAndSet operation (needed for sherpa tablet names' uniqid field) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1158 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 22:36:48 +00:00
sears
29d482cffe add missing includes to check_util.h 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1088 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:20 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
645138d928 remove sync() from unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1001 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 20:38:04 +00:00
sears
7bed4b67ba fix a bunch of 64-bit integer signedness bugs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@940 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-16 21:43:21 +00:00
sears
eb8b57378a laying the groundwork for snowshoveling; as of this commit, pointers and null padding happens in an order that should let us query incomplete tree components. We should only need short page latches at this point (so that we do not look at partially written keys/values) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@931 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-14 22:42:26 +00:00
sears
aa7f7189d6 admission control *almost* works; c1 sometimes blocks for a long time when a new c2 is created 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@809 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-26 00:58:17 +00:00
sears
2a1157602a Rewrite merge thread synchronization code, update status messages, implement preliminary (and ineffective) admission control, and force write merged data every megabyte (so that prograess can be tracked by admission control. Also, refactor quite a few things. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@805 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-19 23:42:06 +00:00
sears
7b2560d906 initial merge manager (multi-merge) statistics 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@797 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 23:13:04 +00:00
sears
176a1e804a more encapsulation changes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@794 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 00:57:48 +00:00
sears
151373d09a refactoring; move init_stasis into logtable, put merge stats into their own module 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@792 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 19:21:25 +00:00
sears
2f15673c80 pass sequential I/O hints to stasis 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@773 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-12 20:56:54 +00:00
sears
e1c937a602 Refactoring of logtable. Cleaned a bunch of includes, and logtable is now a template. This is in preparation for the new merge policy 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@698 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-17 21:51:26 +00:00
sears
28433bf336 remove logiterators.h/cpp and stop #including datapage.cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@690 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-13 00:41:37 +00:00
sears
c8c48a188d more refactoring; all iterators now live in their respective tree components, fixed some hardcoded parameters; encapsulate allocators inside of diskTreeComponent 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@689 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-13 00:05:06 +00:00
sears
5f0d5c4f97 use RegionAllocator for InternalNodes and DataPages 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@685 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 23:17:03 +00:00
sears
8d7199676c made diskTreeComponent more object oriented; many of its functions are now private 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@684 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 22:18:55 +00:00
sears
5d533a7939 DataPage::RegionAllocator -> RegionAllocator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@682 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 20:29:03 +00:00
sears
54e73ab803 move the internal node region allocator stuff into diskTreeComponent::internalNode 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@681 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 20:23:15 +00:00
sears
01c65d7a3c diskTreeComponent -> diskTreeComponent::internalNodes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@680 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 19:47:12 +00:00
sears
1f2f4c745b more iterator renaming; also, getnext() -> next_callerFrees() 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@679 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 19:02:54 +00:00
sears
de7984b05a create memTreeComponent class. Moved memTreeIterator, rbtree_t and tearDownTree into it (still need to move changableMemTreeIterator into it...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@673 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 01:42:23 +00:00
sears
220d58bd7c DataPage::RecordIterator -> DataPage::iterator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@672 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 00:14:24 +00:00
sears
452ee97442 diskTreeComponentIterator -> diskTreeComponent::iterator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@671 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-09 00:09:38 +00:00
sears
a79c7d1c46 check_tcpclient now examines the output of its iterator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@668 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-05 23:23:43 +00:00
sears
06fa7999cb c++-ify diskTreeComponentIterator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@667 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-05 22:21:44 +00:00
sears
2d44b6beed Fix another datapage leak in unit test. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@649 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-27 00:33:12 +00:00
sears
9c2c41c30c fix build on non-stasis machines (workaround fact that check_tcpclient incorrectly requires stasis...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@644 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-26 18:51:12 +00:00
sears
c07c1486ef fix remaining regressed unit test; check_datapage.cpp was leaking a datapage, then running off the end of its buffer. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@631 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 20:17:04 +00:00
sears
52dcae159e fix 3 of 4 unit test regressions 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@628 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 19:03:20 +00:00
sears
6bc4bc1f82 build the client library even if stasis is not present 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@604 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-21 17:02:14 +00:00
sears
63a14d8509 network iterator API is now working (though it could use a better test suite) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@601 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-20 01:18:39 +00:00
sears
57c9afc8d8 rename logtree to diskTreeComponent 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@591 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:59:14 +00:00
sears
ed9d4fc377 Encapsulate allocator state inside logtree. Remove redundant information from merger_args. Rewrite merge threads. This seems to fix the double free issues, and gets rid of disk space leaks. It's probably leaking ram, and does not keep the logstore headers up to date. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@587 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-18 23:31:57 +00:00
sears
69dafd9a55 api cleanups; it is now more difficult to get confused and pass the wrong allocator into a datapage 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@564 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-16 01:01:52 +00:00
sears
6f968fd825 Rewrote datapage region allocator. It no longer scribbles all over itself during merges, but it tries to double free datapage regions. Committing with datapage region reclamation disabled for now. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@562 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-15 23:02:01 +00:00
sears
d769bc0de6 fix compiler warnings. This commit makes use of an API change, and requires Stasis >= r1352 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@561 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-11 21:38:05 +00:00
sears
f2df52319d Rewrote datapage. There is a feature regression: datapages do not try to extend themselves when they encounter a large record, and there is a 1:1 relationship between datapages and regions. A later revision will change the interface with the region allocator in order to address this. This will establish a 1 region -> n datapage relationship. There used to be an n:m mapping. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@560 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-11 20:04:42 +00:00
sears
940a6da6fe Reworked memory allocation and network protocol. This gargantuan commit also reduces the default size of C0, and removes quite a bit of redundant logic and error handling. 
The tests now pass, except that check_merge never terminates (it takes too long) and check_mergelarge still is not passing.

For better luck running this version of the code, turn off stasis' concurrent buffer manager.  We're doing something bad that leads to deadlocks with the concurrent buffer 
manager.  Another (the same?) bug less-frequently leads to page corruption with the old stasis buffer manager.




git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@556 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-10 21:49:50 +00:00
sears
1725b6569d backport client code from ydht load generator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@543 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-02 23:50:16 +00:00