Commit graph

210 commits

Author SHA1 Message Date
michim
ff6e081ca1 use htonl. fixed a typo in daemontools run script.
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2552 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-13 06:53:39 +00:00
michim
88c718c828 commented out timestamp hack
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2550 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 21:08:41 +00:00
michim
c9695d4fd1 commented out timestamp hack
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2549 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 21:06:31 +00:00
michim
49b4d4ebdb added some option to daemontools run script.
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2548 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 21:03:09 +00:00
michim
129342b42c added remove().
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2547 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 20:56:04 +00:00
michim
5291a25dce added packages and conf directories. added update and scan operations.
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2546 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 20:41:18 +00:00
michim
abda0606de moving to thrift
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2545 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-12 00:30:02 +00:00
michim
630c16d12e moving to thrift.
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2543 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-11 22:27:48 +00:00
sears
005ee8a776 fix memory leak, add asserts
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2531 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-02 22:10:23 +00:00
sears
72ecdea59e disable spammy printfs, change assert so that it prints extra debugging info, then ignores error
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2530 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-02 22:09:59 +00:00
sears
bb3e768df7 remove line that accidentally disabled strippedkeylen()
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2529 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-05-02 22:05:32 +00:00
sears
ae7c827ab5 add hacky garbage collection mechanism
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2484 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-28 21:56:25 +00:00
sears
c910a7af8f fix latching bug in scan
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2483 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-28 20:18:54 +00:00
sears
e81b8522d7 fix memory leaks
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2457 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-25 15:28:45 +00:00
sears
36790f6c2c add comment
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2455 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-24 21:09:54 +00:00
sears
5010688123 fix compiler warning on 32 bit architectures;
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2453 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-23 18:53:38 +00:00
sears
5be84f59e8 add support for early tombstone dropping, extend datatuple format to support timestamps
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2451 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 22:54:49 +00:00
sears
9672bc4f48 add commented out debugging code
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2450 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 17:41:13 +00:00
sears
5343f3021e fix broken merge logic. it was sometimes dropping live tuples, and never dropping tombstones...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2449 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 17:40:37 +00:00
sears
4f74dfd45f documentation cleanup
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2448 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 17:39:52 +00:00
sears
38f1793eaa fix argument parsing typo
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2447 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 17:39:28 +00:00
sears
ab9cfd1443 add support for timestamps to datatuple comparator
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2446 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-22 17:38:18 +00:00
sears
65efc768a4 forgot to commit change_log_mode utility
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2438 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-21 00:53:23 +00:00
sears
c42a275ab4 fix shutdown bug; add debugging utility that allows the log to be enabled / batched / disabled at runtime
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2437 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-21 00:28:05 +00:00
sears
8135bbcc2e misc log cleanups; add command line parameter to newserver to control log mode
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2435 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-20 21:51:04 +00:00
sears
0508f6cb40 initial wal implementation
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2434 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-04-20 20:17:26 +00:00
sears
e5058da393 fix merge deadlock; port to fixed stasis buffer manager api; enable new buffer manager by default
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@2330 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-03-25 20:05:49 +00:00
sears
0a0646e75e ignore sigpipe
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1713 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-25 02:07:16 +00:00
sears
95c7290ac6 import eclipse project goo
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1712 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-25 02:06:42 +00:00
sears
f4936551bc optimize datatuple in-memory layout
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1590 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-11 22:08:10 +00:00
sears
975410845f add empirically determined estimators of c0 size, and pessimistic ones that are the default
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1589 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-11 22:07:39 +00:00
sears
abe7e36640 add comment explaining how to enable stlslab froms stasis
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1588 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-11 22:06:18 +00:00
sears
e674f4459a whitespace
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1587 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-11 22:05:52 +00:00
sears
dad55b564c Misc statistics cleanup; improved merge cost estimations during backpressure.
Misc statistics cleanups include:
 - fixes for corner case where c0 is empty, and it would divide by zero
 - always compute number of tuples (in / out, but not base)
 - rework amount of memory allocated when --test and --newserver are passed in.



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1582 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2011-01-11 19:24:16 +00:00
sears
42b52a4d17 fix datapage padding behavior; the old padding overhead was ~ 100%; now it is much lower
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1522 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-17 21:16:43 +00:00
sears
83fe4b7703 fix array-out-of-bounds bug in unit test
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1521 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-17 21:15:26 +00:00
sears
a6a0ce3f54 extended stats cleanup, verbose mode, documentation
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1518 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-17 18:48:26 +00:00
sears
052508fb76 remove mergeStats->current_size field, which is derivable from other fields in mergeStats
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1508 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-15 00:15:59 +00:00
sears
cd27f50baf remove ltable->tsize and ltable->tree_bytes fields
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1507 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-14 23:06:21 +00:00
sears
7e2d12f6ba fix typo in table of response code constants
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1505 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-14 21:43:50 +00:00
sears
73e72b47b2 more stats cleanups
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1493 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-14 01:49:23 +00:00
sears
43425128ba clean up initialization / marshalling code. Lets some of the public mergeStat fields to be protected
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1490 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-14 00:06:32 +00:00
sears
eb4446f409 readability fix: fields in mergeStats that are only used for computing extended stats for pretty print are now prefixed with "stats_", and can be optionally disabled at compile time
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1489 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-13 22:27:13 +00:00
sears
7e44641799 move c1-c2 backpresure to linear backpressure model; removes a few hundred lines of fussy code, and should enable further simplifiactions
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1488 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-13 21:17:35 +00:00
sears
347a4126f3 remove the NO_SNOWSHOVEL c-preprocessor stuff. There is no way it was working anyway...
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1480 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-11 01:04:33 +00:00
sears
a00531ae6d simplified merge_scheduler
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-11 00:51:19 +00:00
sears
8dddc7e168 remove line from unit test that had no effect, but broke compilation
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1478 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-10 20:42:57 +00:00
sears
a8e905c0d6 fix deadlock on shutdown; be more stingy with bloom filter RAM with small indexes, add bloom filter to c2, default back to old buffer manager
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1468 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-08 19:49:13 +00:00
sears
79936da79b loosen some scheduling parameters
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1467 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-08 19:46:55 +00:00
sears
e9b68056b8 whitespace
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1466 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
2010-12-08 19:18:05 +00:00