greg/stasis-bLSM
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
233 commits 1 branch 0 tags 670 KiB
Commit graph

233 commits

This branch
This branch
All branches
Author SHA1 Message Date
sears
960ff041e2 remove old comments 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1055 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-24 00:40:48 +00:00
sears
6fd7dd0523 cleaned up some code, moved pretty_print into its own thread, and fixed some buggy stats computations 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1054 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 23:28:29 +00:00
sears
98b88a227b fixed some statistics computations. They are still wrong. It crashes for an unknown reason. However, this version sometimes breaks 10K op/s for loads, and 1K op/s for workloada 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1047 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-23 17:11:31 +00:00
sears
bcea21c06a fixed memory bug in client scams; better debugging status during merge backpressure 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1046 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:30:12 +00:00
sears
07b35f4e20 scattered, and poorly tested fixes. finer grained latching for tree nodes; throttle c0-c1 merge when c0 is small; fix delta computations for c0 throttling; disable short pauses for c1-c2 merge. this gets the bulk load throughput up to 8500, but workloada is still slow 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1045 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-21 03:09:18 +00:00
sears
0d471bb912 fix typo that leads to double frees 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1018 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 18:06:59 +00:00
sears
b894cebaf7 add memTreeComponent::batchedRevalidatingIterator; amortize cost of red black latching during merge 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1017 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-18 17:29:25 +00:00
sears
1e487bbc54 re-enable old merge policy for c1-c2 merger; implement shutdown (sort of) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1006 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-17 21:23:39 +00:00
sears
b4f80aaa7d added shutdown utility 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1002 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 22:11:19 +00:00
sears
645138d928 remove sync() from unit test 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@1001 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-16 20:38:04 +00:00
sears
ff309a6fbf fix latch/pin leak 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@990 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-12 01:55:30 +00:00
sears
a5cc4e5b53 fix a deadlock (double setting thread_fd = -1 in race at startup), lazily spawn new worker threads. This will save some ram, and will lead to shorter server thread listings in gdb 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@987 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 23:27:39 +00:00
sears
bded48a306 add new server implementation that uses FILE to buffer I/O, and handles a smaller, fixed number of connections -- leads to an almost 4x speedup on no-ops on nehalem 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@986 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 22:46:55 +00:00
sears
569b46363b moved most of the request dispatching logic from logserver.cpp to requestDispatch.cpp 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@985 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-11 17:43:39 +00:00
sears
4a42e4f904 tcpclient now usees fread/fwrite instead of read/write 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@983 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-10 23:56:54 +00:00
sears
2a45c2cda8 add no-op network call, and a simple benchmark that exercises it 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@980 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-09 20:21:41 +00:00
sears
dc75f6d1f0 Implemented snowshoveling, with caveats. 
This commit includes concurrency control for datapages, the new backpressure for the c0-c1 merger, and (untested) support for defining NO_SNOWSHOVEL, and using the old approach.

There are a number of important limitations:
 - statistics gathering is somewhat messed up
 - the c0-c1 merge thread polls for new data
 - on a sequential write, if the writers stall, the merge thread could empty memory and initiate an unnecessary merge.  Instead, we should never finish a merge if c0 is less than N% full
 - this commit introduces yet another batch of nasty mutex contention issues



git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@955 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:43:46 +00:00
sears
00d6cd2440 added destructor to diskTreeComponent (fixes file handle / memory leak) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@954 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:39:31 +00:00
sears
7a1b2f9687 revert change that silenced g++ warning. the change broke ancient g++ 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@953 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-08-05 17:23:12 +00:00
sears
29555bd6f6 silence compiler warning 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@941 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-20 19:12:47 +00:00
sears
7bed4b67ba fix a bunch of 64-bit integer signedness bugs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@940 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-16 21:43:21 +00:00
sears
eb8b57378a laying the groundwork for snowshoveling; as of this commit, pointers and null padding happens in an order that should let us query incomplete tree components. We should only need short page latches at this point (so that we do not look at partially written keys/values) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@931 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-14 22:42:26 +00:00
sears
9dcb624649 improved bps estimation; add "minisleeps" to update_progress; they do not belong there long term, but this is better than nothing 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@929 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-07-14 21:46:27 +00:00
sears
cf6c6e5955 incease UPDATE_PROGRESS_DELTA by 10x, and FORCE_INTERVAL by 2.5x; tick no longer grabs a writelock on header_mut, and instead uses its own mutex 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@870 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 22:59:05 +00:00
sears
fded91fec6 two unrelated changes: release rb_mut before getting header_mut on read path (breaks delta tuples), and implement first-cut support for pulling in statistics when reopening logstores (required for correct recovery) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@868 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-21 20:03:05 +00:00
sears
118ae94a9b cleaned up comments; upped force_interval to 10mb 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@864 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 23:52:17 +00:00
sears
1c02dc1281 add overshoot2 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@863 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 23:44:48 +00:00
sears
8c35a1265a fix bug in overshoot_fudge computation; reduce number of knobs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@860 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 23:00:23 +00:00
sears
eda7c0fd12 enable force interval; play with some knobs 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@854 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 02:07:55 +00:00
sears
2530009ec0 fix deadlock; do not clear need_tick bit until overshoot is less than zero; tweak constants accordingly (fewer mutex acquistions! less blockage!) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@852 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-18 00:06:46 +00:00
sears
0a2c4796a8 removed bad assert 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@843 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-17 05:05:35 +00:00
sears
0fb644640a improved concurrency for merger. not happy with worst case write latency on my dev machine anymore (seems to be less than 2.5 seconds, up from less than 1). 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@842 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-17 04:49:19 +00:00
sears
0ecd7491c7 fix minor stats bugs; pull most stats computation into update_progress. The idea is that update_progress and tick, which are easily amortized, are the only stats things that need to grab latches 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@827 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-05 00:41:52 +00:00
sears
fe7ce3a785 tweak some constants to silence latency sla warnings 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@825 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-03 00:12:31 +00:00
sears
eda514ddaa amortize the computation of statistics (but not mutex acuqisitions) across many tuple insertions. Also, refactor stats stuff with latch contention in mind (but it still grabs too many latches...). This commit causes check_tcpclient to emit warnings on my machine. Not sure if it will happen on production hardware 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@824 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-06-02 21:47:58 +00:00
sears
fcf0dcca99 cleanup / reimplement some of the statistics calculations. dial back periodically_force 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@821 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 21:20:26 +00:00
sears
dca4fbe594 revert server thread count (it was set to 5 for debugging, then accidentally committed...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@820 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 18:31:34 +00:00
sears
4050475de9 more bugfixes / cleanup. latching is probably overly conservative at this point, but ycsb bulkload finally seems to work 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@819 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-28 01:29:10 +00:00
sears
d981d91dfa numerous bugfixes; rate limit progress meter display 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@816 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 23:15:24 +00:00
sears
6eecf1557a restore support for concurrent disk reads; logserver was accidentally using a findTuple function that assumed diffs. This commit uses rwlc, and requires Stasis -r1393 or better 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@814 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-27 01:49:27 +00:00
sears
0e375c005b more merge tweaks 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@813 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-26 23:53:21 +00:00
sears
aa7f7189d6 admission control *almost* works; c1 sometimes blocks for a long time when a new c2 is created 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@809 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-26 00:58:17 +00:00
sears
c50c14d259 admission control is now behaving properly (but there are still no concurrent writes, or bounds on how long admission control throttles a given writer) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@806 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-21 23:43:17 +00:00
sears
2a1157602a Rewrite merge thread synchronization code, update status messages, implement preliminary (and ineffective) admission control, and force write merged data every megabyte (so that prograess can be tracked by admission control. Also, refactor quite a few things. 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@805 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-19 23:42:06 +00:00
sears
1adb980555 added throttling policy and better stats. the throttling makes a merge policy bug painfully obvious (committing because i plan to do some refactoring...) 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@802 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-05-12 22:16:41 +00:00
sears
7b2560d906 initial merge manager (multi-merge) statistics 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@797 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 23:13:04 +00:00
sears
ce48882946 oops; forgot to add mergeStats.h; finished encapsuating its state 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@795 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 01:03:56 +00:00
sears
176a1e804a more encapsulation changes 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@794 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-29 00:57:48 +00:00
sears
ded9cc2585 fixed rounding errors in statistics written to stdout; whitespace cleanup; more encapsulation 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@793 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 21:29:15 +00:00
sears
151373d09a refactoring; move init_stasis into logtable, put merge stats into their own module 
git-svn-id: svn+ssh://svn.corp.yahoo.com/yahoo/yrl/labs/pnuts/code/logstore@792 8dad8b1f-cf64-0410-95b6-bcf113ffbcfe
 2010-04-28 19:21:25 +00:00