greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
178 commits 1 branch 0 tags 670 KiB
Commit graph

178 commits

This branch
This branch
All branches
Author SHA1 Message Date
sears
e674f4459a whitespace 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1587 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-01-11 22:05:52 +00:00
sears
dad55b564c Misc statistics cleanup; improved merge cost estimations during backpressure. 
Misc statistics cleanups include:
 - fixes for corner case where c0 is empty, and it would divide by zero
 - always compute number of tuples (in / out, but not base)
 - rework amount of memory allocated when --test and --newserver are passed in.



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1582 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2011-01-11 19:24:16 +00:00
sears
42b52a4d17 fix datapage padding behavior; the old padding overhead was ~ 100%; now it is much lower 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1522 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-17 21:16:43 +00:00
sears
83fe4b7703 fix array-out-of-bounds bug in unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1521 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-17 21:15:26 +00:00
sears
a6a0ce3f54 extended stats cleanup, verbose mode, documentation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1518 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-17 18:48:26 +00:00
sears
052508fb76 remove mergeStats->current_size field, which is derivable from other fields in mergeStats 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1508 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-15 00:15:59 +00:00
sears
cd27f50baf remove ltable->tsize and ltable->tree_bytes fields 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1507 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 23:06:21 +00:00
sears
7e2d12f6ba fix typo in table of response code constants 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1505 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 21:43:50 +00:00
sears
73e72b47b2 more stats cleanups 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1493 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 01:49:23 +00:00
sears
43425128ba clean up initialization / marshalling code. Lets some of the public mergeStat fields to be protected 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1490 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-14 00:06:32 +00:00
sears
eb4446f409 readability fix: fields in mergeStats that are only used for computing extended stats for pretty print are now prefixed with "stats_", and can be optionally disabled at compile time 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1489 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-13 22:27:13 +00:00
sears
7e44641799 move c1-c2 backpresure to linear backpressure model; removes a few hundred lines of fussy code, and should enable further simplifiactions 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1488 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-13 21:17:35 +00:00
sears
347a4126f3 remove the NO_SNOWSHOVEL c-preprocessor stuff. There is no way it was working anyway... 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1480 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 01:04:33 +00:00
sears
a00531ae6d simplified merge_scheduler 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 00:51:19 +00:00
sears
8dddc7e168 remove line from unit test that had no effect, but broke compilation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1478 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-10 20:42:57 +00:00
sears
a8e905c0d6 fix deadlock on shutdown; be more stingy with bloom filter RAM with small indexes, add bloom filter to c2, default back to old buffer manager 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1468 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:49:13 +00:00
sears
79936da79b loosen some scheduling parameters 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1467 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:46:55 +00:00
sears
e9b68056b8 whitespace 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1466 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:18:05 +00:00
sears
fbdc900623 fix #include mistake 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1239 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 18:27:56 +00:00
sears
ccf3895595 conservatively make the bloom filter really big (should behave well down to 100 byte keys, waste ram otherwise...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1238 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:55:29 +00:00
sears
31cafbef6c get rid of -Wno-variadic-macros since it suddenly broke compilation for some reason 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1237 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:54:27 +00:00
sears
8936df750a fix includes to avoid linker errors 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1236 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:52:32 +00:00
sears
0c5700257c re-re-re-...re-disable new buffer manager optimizations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1235 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:50:39 +00:00
sears
b9dd11a7e4 reduce default datapage_size to 1 (4096) to avoid useless seeks for small values 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1234 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:23:31 +00:00
sears
f9a1f411e1 initial checkin of bloom filter logic for lsm trees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1233 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 23:05:45 +00:00
sears
3619e86271 initial checkin of bloom filter 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1232 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 18:06:19 +00:00
sears
6eaf5da8de port to newer gcc/g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1231 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-02 12:52:41 +00:00
sears
4c0e8a7186 change cerr to log4cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1204 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-29 17:58:34 +00:00
sears
e5518eac26 enable debugging 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1178 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:26 +00:00
sears
1c8c0fe0da switch default buffer manager to concurrentBufferManager (AGAIN) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1177 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:07 +00:00
sears
392a097e44 add atomic testAndSet operation (needed for sherpa tablet names' uniqid field) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1158 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 22:36:48 +00:00
sears
9ca7d78b06 add some error checking. testTablet is closer to passing 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1157 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 21:13:43 +00:00
sears
5bd187c481 misc cleanups, removed old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1152 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:53:30 +00:00
sears
3e8ffcbd7f Sherpa 1.10 storage/handler/unit test/ now runs to completion with the following failures: 
make: *** [testCachedMetadataStorerun] Error 255
make: *** [testGetCopyStatusrun] Error 255
make: *** [testGetLbStatsrun] Error 255
make: *** [testGetRecordFormatterrun] Error 255
make: *** [testGetSplitPointrun] Error 255
make: *** [testGetTabletMetadatarun] Error 255
make: *** [testKillRecordrun] Error 255
make: *** [testMySQLPersistentrun] Error 255
make: *** [testMySQLUpgrade16to18run] Error 255
make: *** [testPingrun] Error 255
make: *** [testRecordUtilsrun] Error 255
make: *** [testResetRecordMetadatarun] Error 255
make: *** [testScanTabletrun] Error 255
make: *** [testTabletrun] Error 255



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1151 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:37:50 +00:00
sears
2eda51e176 partially working sherpa 1.10 backend with .so support 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1150 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 21:53:51 +00:00
sears
4eb2bb852c add a #define to turn TCP_NODELAY on/off 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1099 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:03:13 +00:00
sears
10556c6df2 add megabytes to status output 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1098 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:51 +00:00
sears
46eddd1269 amortize latches during scan 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1097 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:24 +00:00
sears
b6e7bf214b add copy database utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1091 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:57 +00:00
sears
105936eeba partially roll-back last commit, which contained some changes that turned requestDispatch->bulk_insert into a no-op for benchmarking 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1090 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:27 +00:00
sears
b1e5e3ad78 fix minor memory leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1089 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:52 +00:00
sears
29d482cffe add missing includes to check_util.h 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1088 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:20 +00:00
sears
b769e1866f fix deadlock and memory leak in network protocols 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1087 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:39:49 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
f8248240af calculate c0->current_size more eagerly 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1058 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-25 16:36:17 +00:00
sears
960ff041e2 remove old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1055 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-24 00:40:48 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
98b88a227b fixed some statistics computations. They are still wrong. It crashes for an unknown reason. However, this version sometimes breaks 10K op/s for loads, and 1K op/s for workloada 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1047 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 17:11:31 +00:00
sears
bcea21c06a fixed memory bug in client scams; better debugging status during merge backpressure 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1046 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:30:12 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00