greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
51 commits 1 branch 0 tags 670 KiB
Commit graph

51 commits

This branch
This branch
All branches
Author SHA1 Message Date
sears
632afe6d9c cleanup statistics gathering and reporting. 
constify inputs to merge functions (so they can't call freetuple())


git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@662 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-05 19:07:47 +00:00
sears
f258ce1a3f adding a few overlooked files 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@657 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-03 23:21:08 +00:00
sears
1243d999a5 Hacky histogram generation; API's for client-side iterators. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@656 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-03 23:20:05 +00:00
sears
747f4df1f3 Reopening existing trees now seems to work (no thorough tests yet). Added some network opcodes for statistics and debugging (not all are implemented yet). Reduce verbosity of server and client library. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@654 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-03-01 21:26:07 +00:00
sears
27720dddbe Implemented (but did not test) header update. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@650 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-27 00:35:13 +00:00
sears
2d44b6beed Fix another datapage leak in unit test. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@649 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-27 00:33:12 +00:00
sears
20dc06280b Initial checkin of sherpa backend. These files probably don't belong here in the long term, but this is expedient. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@645 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-26 18:52:25 +00:00
sears
9c2c41c30c fix build on non-stasis machines (workaround fact that check_tcpclient incorrectly requires stasis...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@644 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-26 18:51:12 +00:00
sears
51df61f8e9 logtableIterator now drops tombstone tuples by default 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@642 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-26 17:45:31 +00:00
sears
c07c1486ef fix remaining regressed unit test; check_datapage.cpp was leaking a datapage, then running off the end of its buffer. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@631 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 20:17:04 +00:00
sears
52dcae159e fix 3 of 4 unit test regressions 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@628 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 19:03:20 +00:00
sears
cb614dff1b fix nasty bug where diskTreeComponent (perhaps) drops tuples. With the recently added paranoid checks, this fix should do nothing worse than trigger assertion failures (as opposed to silently losing data...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@627 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 19:02:09 +00:00
sears
27e6f0b0f4 make tuple argument to RecordIterator optional 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@626 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 19:00:27 +00:00
sears
7c2397340c Fixed a bunch of iterator bugs and racy merges. Added drop_database utility. 
Bugfixes:
 - Atomically deallocate regions and update the logstore object
 - Proactively invalidate iterators after each merge (before, it would simply set a not-valid bit.  This doesn't work because iterators hold page pins, which breaks force-writes)
 - Clarify semantics of opening iterators mid-stream:  All calls now return iterators that return first key >= the requested one.  revalidate() needs the key > the requested one, so it calls peek(), then (if necessary) getnext().
 - Add asserts to check that the header is latched at update, and that tuples returned by iterators are strictly monotonically increasing'
 - Improve error handling in network.h  We still get (and terminate on) SIGPIPE.

Refactoring:
 - Add dispatch function to network.h.



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@620 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-25 01:29:32 +00:00
sears
abce19c67e Fixed server hang. It used to sleep forever inside select if all the clients hung. Now, it uses the self pipe trick to wake itself up. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@614 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-23 23:54:44 +00:00
sears
6a2d6a1cd4 Generalized comparator, fixed some bugs in record length handling. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@613 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-23 17:05:47 +00:00
sears
a0a9d726fd install.sh script for yroots 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@605 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-21 17:04:30 +00:00
sears
6bc4bc1f82 build the client library even if stasis is not present 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@604 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-21 17:02:14 +00:00
sears
63a14d8509 network iterator API is now working (though it could use a better test suite) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@601 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-20 01:18:39 +00:00
sears
57c9afc8d8 rename logtree to diskTreeComponent 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@591 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:59:14 +00:00
sears
2b82a62a34 broke logstore.cpp into two pieces 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@590 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:52:21 +00:00
sears
7738b67bc0 Implemented network iterator protocol. It hasn't been tested or hooked up to anything because logstore iterators don't exist yet. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@589 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-19 00:34:41 +00:00
sears
ed9d4fc377 Encapsulate allocator state inside logtree. Remove redundant information from merger_args. Rewrite merge threads. This seems to fix the double free issues, and gets rid of disk space leaks. It's probably leaking ram, and does not keep the logstore headers up to date. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@587 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-18 23:31:57 +00:00
sears
ec76d1c244 fix build on RHEL 4.8 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@586 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-18 23:25:23 +00:00
sears
ec747781ac partially remove merger_args structure 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@579 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-17 23:38:31 +00:00
sears
5032e77b6c more allocator cleanups (still leaks space). Added dump_blockmap utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@578 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-17 22:11:22 +00:00
sears
e2ea48d233 remove a bunch of whitespace / comments. clean up network thread. add constant for dumping the block map 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@570 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-16 23:15:29 +00:00
sears
478d394365 remove comments/whitespace 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@565 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-16 01:04:46 +00:00
sears
69dafd9a55 api cleanups; it is now more difficult to get confused and pass the wrong allocator into a datapage 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@564 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-16 01:01:52 +00:00
sears
91273d2f43 removed alloc state from merger_args; space reclamation now seems to be working, modulo ArrayList leaks 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@563 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-16 00:44:13 +00:00
sears
6f968fd825 Rewrote datapage region allocator. It no longer scribbles all over itself during merges, but it tries to double free datapage regions. Committing with datapage region reclamation disabled for now. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@562 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-15 23:02:01 +00:00
sears
d769bc0de6 fix compiler warnings. This commit makes use of an API change, and requires Stasis >= r1352 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@561 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-11 21:38:05 +00:00
sears
f2df52319d Rewrote datapage. There is a feature regression: datapages do not try to extend themselves when they encounter a large record, and there is a 1:1 relationship between datapages and regions. A later revision will change the interface with the region allocator in order to address this. This will establish a 1 region -> n datapage relationship. There used to be an n:m mapping. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@560 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-11 20:04:42 +00:00
sears
940a6da6fe Reworked memory allocation and network protocol. This gargantuan commit also reduces the default size of C0, and removes quite a bit of redundant logic and error handling. 
The tests now pass, except that check_merge never terminates (it takes too long) and check_mergelarge still is not passing.

For better luck running this version of the code, turn off stasis' concurrent buffer manager.  We're doing something bad that leads to deadlocks with the concurrent buffer 
manager.  Another (the same?) bug less-frequently leads to page corruption with the old stasis buffer manager.




git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@556 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-10 21:49:50 +00:00
sears
3c297d1a66 fix iterator bug 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@550 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-04 22:19:42 +00:00
sears
d5b4106b7e silence verbose messages 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@549 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-04 22:19:13 +00:00
sears
1725b6569d backport client code from ydht load generator 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@543 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-02 23:50:16 +00:00
sears
3cbcb1e274 check_tcpclient now works (sort of...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@541 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-02-02 22:50:09 +00:00
sears
094a0d41ee removed some more redundant code by unifying merge_iterators 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@532 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-28 02:20:49 +00:00
sears
23359c2570 fix more compiler warnings 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@531 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-27 23:34:33 +00:00
sears
5135bb5924 clean up includes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@530 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-27 23:15:47 +00:00
sears
616c028b17 move send_tuple into utils, fix warnings 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@529 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-27 22:29:12 +00:00
sears
2f562a175a remove more duplicated code 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@528 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 20:19:36 +00:00
sears
cead29f0e9 fix compiler warning; remove dead code 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@527 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 20:03:27 +00:00
sears
38f2cb3897 de-duplicated some cut and paste code 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@526 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 03:37:07 +00:00
sears
2fcfafb589 remove more crud 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@525 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 00:54:38 +00:00
sears
b820aaa3fb commit cmakelists file, move tests into subdirectory 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@524 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 00:50:01 +00:00
sears
78f9ad12c1 removed dead files, switch to CMake 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@523 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-26 00:37:05 +00:00
sears
648a725920 found + documented bug (in the middle of big cleanup, so did not fix it) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@522 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-25 23:03:03 +00:00
sears
18cb8848f1 port + test with stasis r1351 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@521 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-01-23 02:16:06 +00:00