greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
165 commits 1 branch 0 tags 670 KiB
Commit graph

165 commits

This branch
This branch
All branches
Author SHA1 Message Date
sears
a00531ae6d simplified merge_scheduler 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1479 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-11 00:51:19 +00:00
sears
8dddc7e168 remove line from unit test that had no effect, but broke compilation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1478 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-10 20:42:57 +00:00
sears
a8e905c0d6 fix deadlock on shutdown; be more stingy with bloom filter RAM with small indexes, add bloom filter to c2, default back to old buffer manager 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1468 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:49:13 +00:00
sears
79936da79b loosen some scheduling parameters 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1467 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:46:55 +00:00
sears
e9b68056b8 whitespace 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1466 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-12-08 19:18:05 +00:00
sears
fbdc900623 fix #include mistake 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1239 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 18:27:56 +00:00
sears
ccf3895595 conservatively make the bloom filter really big (should behave well down to 100 byte keys, waste ram otherwise...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1238 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:55:29 +00:00
sears
31cafbef6c get rid of -Wno-variadic-macros since it suddenly broke compilation for some reason 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1237 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:54:27 +00:00
sears
8936df750a fix includes to avoid linker errors 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1236 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:52:32 +00:00
sears
0c5700257c re-re-re-...re-disable new buffer manager optimizations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1235 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:50:39 +00:00
sears
b9dd11a7e4 reduce default datapage_size to 1 (4096) to avoid useless seeks for small values 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1234 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:23:31 +00:00
sears
f9a1f411e1 initial checkin of bloom filter logic for lsm trees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1233 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 23:05:45 +00:00
sears
3619e86271 initial checkin of bloom filter 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1232 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 18:06:19 +00:00
sears
6eaf5da8de port to newer gcc/g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1231 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-02 12:52:41 +00:00
sears
4c0e8a7186 change cerr to log4cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1204 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-29 17:58:34 +00:00
sears
e5518eac26 enable debugging 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1178 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:26 +00:00
sears
1c8c0fe0da switch default buffer manager to concurrentBufferManager (AGAIN) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1177 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:07 +00:00
sears
392a097e44 add atomic testAndSet operation (needed for sherpa tablet names' uniqid field) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1158 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 22:36:48 +00:00
sears
9ca7d78b06 add some error checking. testTablet is closer to passing 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1157 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 21:13:43 +00:00
sears
5bd187c481 misc cleanups, removed old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1152 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:53:30 +00:00
sears
3e8ffcbd7f Sherpa 1.10 storage/handler/unit test/ now runs to completion with the following failures: 
make: *** [testCachedMetadataStorerun] Error 255
make: *** [testGetCopyStatusrun] Error 255
make: *** [testGetLbStatsrun] Error 255
make: *** [testGetRecordFormatterrun] Error 255
make: *** [testGetSplitPointrun] Error 255
make: *** [testGetTabletMetadatarun] Error 255
make: *** [testKillRecordrun] Error 255
make: *** [testMySQLPersistentrun] Error 255
make: *** [testMySQLUpgrade16to18run] Error 255
make: *** [testPingrun] Error 255
make: *** [testRecordUtilsrun] Error 255
make: *** [testResetRecordMetadatarun] Error 255
make: *** [testScanTabletrun] Error 255
make: *** [testTabletrun] Error 255



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1151 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:37:50 +00:00
sears
2eda51e176 partially working sherpa 1.10 backend with .so support 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1150 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 21:53:51 +00:00
sears
4eb2bb852c add a #define to turn TCP_NODELAY on/off 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1099 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:03:13 +00:00
sears
10556c6df2 add megabytes to status output 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1098 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:51 +00:00
sears
46eddd1269 amortize latches during scan 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1097 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:24 +00:00
sears
b6e7bf214b add copy database utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1091 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:57 +00:00
sears
105936eeba partially roll-back last commit, which contained some changes that turned requestDispatch->bulk_insert into a no-op for benchmarking 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1090 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:27 +00:00
sears
b1e5e3ad78 fix minor memory leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1089 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:52 +00:00
sears
29d482cffe add missing includes to check_util.h 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1088 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:20 +00:00
sears
b769e1866f fix deadlock and memory leak in network protocols 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1087 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:39:49 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
f8248240af calculate c0->current_size more eagerly 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1058 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-25 16:36:17 +00:00
sears
960ff041e2 remove old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1055 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-24 00:40:48 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
98b88a227b fixed some statistics computations. They are still wrong. It crashes for an unknown reason. However, this version sometimes breaks 10K op/s for loads, and 1K op/s for workloada 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1047 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 17:11:31 +00:00
sears
bcea21c06a fixed memory bug in client scams; better debugging status during merge backpressure 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1046 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:30:12 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00
sears
0d471bb912 fix typo that leads to double frees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1018 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 18:06:59 +00:00
sears
b894cebaf7 add memTreeComponent::batchedRevalidatingIterator; amortize cost of red black latching during merge 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1017 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 17:29:25 +00:00
sears
1e487bbc54 re-enable old merge policy for c1-c2 merger; implement shutdown (sort of) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1006 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-17 21:23:39 +00:00
sears
b4f80aaa7d added shutdown utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1002 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 22:11:19 +00:00
sears
645138d928 remove sync() from unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1001 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 20:38:04 +00:00
sears
ff309a6fbf fix latch/pin leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@990 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-12 01:55:30 +00:00
sears
a5cc4e5b53 fix a deadlock (double setting thread_fd = -1 in race at startup), lazily spawn new worker threads. This will save some ram, and will lead to shorter server thread listings in gdb 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@987 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 23:27:39 +00:00
sears
bded48a306 add new server implementation that uses FILE to buffer I/O, and handles a smaller, fixed number of connections -- leads to an almost 4x speedup on no-ops on nehalem 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@986 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 22:46:55 +00:00
sears
569b46363b moved most of the request dispatching logic from logserver.cpp to requestDispatch.cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@985 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 17:43:39 +00:00
sears
4a42e4f904 tcpclient now usees fread/fwrite instead of read/write 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@983 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-10 23:56:54 +00:00
sears
2a45c2cda8 add no-op network call, and a simple benchmark that exercises it 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@980 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-09 20:21:41 +00:00
sears
dc75f6d1f0 Implemented snowshoveling, with caveats. 
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.

There are a number of important limitations:
 - statistics gathering is somewhat messed up
 - the c0-c1 merge thread polls for new data
 - on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge.  Instead, we should never finish a merge if c0 is less than N% full
 - this commit introduces yet another batch of nasty mutex contention issues



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:43:46 +00:00
sears
00d6cd2440 added destructor to diskTreeComponent (fixes file handle / memory leak) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@954 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:39:31 +00:00