greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
259 commits 1 branch 0 tags 670 KiB
Commit graph

259 commits

This branch
This branch
All branches
Author SHA1 Message Date
sears
ccf3895595 conservatively make the bloom filter really big (should behave well down to 100 byte keys, waste ram otherwise...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1238 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:55:29 +00:00
sears
31cafbef6c get rid of -Wno-variadic-macros since it suddenly broke compilation for some reason 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1237 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:54:27 +00:00
sears
8936df750a fix includes to avoid linker errors 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1236 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:52:32 +00:00
sears
0c5700257c re-re-re-...re-disable new buffer manager optimizations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1235 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:50:39 +00:00
sears
b9dd11a7e4 reduce default datapage_size to 1 (4096) to avoid useless seeks for small values 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1234 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-05 16:23:31 +00:00
sears
f9a1f411e1 initial checkin of bloom filter logic for lsm trees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1233 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 23:05:45 +00:00
sears
3619e86271 initial checkin of bloom filter 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1232 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-03 18:06:19 +00:00
sears
6eaf5da8de port to newer gcc/g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1231 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-10-02 12:52:41 +00:00
sears
4c0e8a7186 change cerr to log4cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1204 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-29 17:58:34 +00:00
sears
e5518eac26 enable debugging 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1178 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:26 +00:00
sears
1c8c0fe0da switch default buffer manager to concurrentBufferManager (AGAIN) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1177 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-22 19:28:07 +00:00
sears
392a097e44 add atomic testAndSet operation (needed for sherpa tablet names' uniqid field) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1158 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 22:36:48 +00:00
sears
9ca7d78b06 add some error checking. testTablet is closer to passing 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1157 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-16 21:13:43 +00:00
sears
5bd187c481 misc cleanups, removed old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1152 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:53:30 +00:00
sears
3e8ffcbd7f Sherpa 1.10 storage/handler/unit test/ now runs to completion with the following failures: 
make: *** [testCachedMetadataStorerun] Error 255
make: *** [testGetCopyStatusrun] Error 255
make: *** [testGetLbStatsrun] Error 255
make: *** [testGetRecordFormatterrun] Error 255
make: *** [testGetSplitPointrun] Error 255
make: *** [testGetTabletMetadatarun] Error 255
make: *** [testKillRecordrun] Error 255
make: *** [testMySQLPersistentrun] Error 255
make: *** [testMySQLUpgrade16to18run] Error 255
make: *** [testPingrun] Error 255
make: *** [testRecordUtilsrun] Error 255
make: *** [testResetRecordMetadatarun] Error 255
make: *** [testScanTabletrun] Error 255
make: *** [testTabletrun] Error 255



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1151 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 22:37:50 +00:00
sears
2eda51e176 partially working sherpa 1.10 backend with .so support 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1150 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-15 21:53:51 +00:00
sears
4eb2bb852c add a #define to turn TCP_NODELAY on/off 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1099 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:03:13 +00:00
sears
10556c6df2 add megabytes to status output 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1098 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:51 +00:00
sears
46eddd1269 amortize latches during scan 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1097 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-02 23:02:24 +00:00
sears
b6e7bf214b add copy database utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1091 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:57 +00:00
sears
105936eeba partially roll-back last commit, which contained some changes that turned requestDispatch->bulk_insert into a no-op for benchmarking 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1090 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:42:27 +00:00
sears
b1e5e3ad78 fix minor memory leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1089 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:52 +00:00
sears
29d482cffe add missing includes to check_util.h 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1088 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:40:20 +00:00
sears
b769e1866f fix deadlock and memory leak in network protocols 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1087 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-09-01 17:39:49 +00:00
sears
a1f3f92cca added bulk-insert APIs, prevent c1-c2 from outrunning c0-c1 by too much, and fix the overshoot computations in merge manager (roughly corresponds to run 20) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1074 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-30 22:22:25 +00:00
sears
f8248240af calculate c0->current_size more eagerly 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1058 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-25 16:36:17 +00:00
sears
960ff041e2 remove old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1055 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-24 00:40:48 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
98b88a227b fixed some statistics computations. They are still wrong. It crashes for an unknown reason. However, this version sometimes breaks 10K op/s for loads, and 1K op/s for workloada 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1047 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 17:11:31 +00:00
sears
bcea21c06a fixed memory bug in client scams; better debugging status during merge backpressure 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1046 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:30:12 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00
sears
0d471bb912 fix typo that leads to double frees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1018 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 18:06:59 +00:00
sears
b894cebaf7 add memTreeComponent::batchedRevalidatingIterator; amortize cost of red black latching during merge 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1017 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 17:29:25 +00:00
sears
1e487bbc54 re-enable old merge policy for c1-c2 merger; implement shutdown (sort of) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1006 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-17 21:23:39 +00:00
sears
b4f80aaa7d added shutdown utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1002 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 22:11:19 +00:00
sears
645138d928 remove sync() from unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1001 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 20:38:04 +00:00
sears
ff309a6fbf fix latch/pin leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@990 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-12 01:55:30 +00:00
sears
a5cc4e5b53 fix a deadlock (double setting thread_fd = -1 in race at startup), lazily spawn new worker threads. This will save some ram, and will lead to shorter server thread listings in gdb 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@987 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 23:27:39 +00:00
sears
bded48a306 add new server implementation that uses FILE to buffer I/O, and handles a smaller, fixed number of connections -- leads to an almost 4x speedup on no-ops on nehalem 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@986 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 22:46:55 +00:00
sears
569b46363b moved most of the request dispatching logic from logserver.cpp to requestDispatch.cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@985 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 17:43:39 +00:00
sears
4a42e4f904 tcpclient now usees fread/fwrite instead of read/write 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@983 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-10 23:56:54 +00:00
sears
2a45c2cda8 add no-op network call, and a simple benchmark that exercises it 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@980 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-09 20:21:41 +00:00
sears
dc75f6d1f0 Implemented snowshoveling, with caveats. 
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.

There are a number of important limitations:
 - statistics gathering is somewhat messed up
 - the c0-c1 merge thread polls for new data
 - on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge.  Instead, we should never finish a merge if c0 is less than N% full
 - this commit introduces yet another batch of nasty mutex contention issues



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:43:46 +00:00
sears
00d6cd2440 added destructor to diskTreeComponent (fixes file handle / memory leak) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@954 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:39:31 +00:00
sears
7a1b2f9687 revert change that silenced g++ warning. the change broke ancient g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@953 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:23:12 +00:00
sears
29555bd6f6 silence compiler warning 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@941 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-20 19:12:47 +00:00
sears
7bed4b67ba fix a bunch of 64-bit integer signedness bugs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@940 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-16 21:43:21 +00:00
sears
eb8b57378a laying the groundwork for snowshoveling; as of this commit, pointers and null padding happens in an order that should let us query incomplete tree components. We should only need short page latches at this point (so that we do not look at partially written keys/values) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@931 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-14 22:42:26 +00:00
sears
9dcb624649 improved bps estimation; add "minisleeps" to update_progress; they do not belong there long term, but this is better than nothing 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@929 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-14 21:46:27 +00:00
sears
cf6c6e5955 incease UPDATE_PROGRESS_DELTA by 10x, and FORCE_INTERVAL by 2.5x; tick no longer grabs a writelock on header_mut, and instead uses its own mutex 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@870 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 22:59:05 +00:00