greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity
364 commits 4 branches 0 tags 2.4 MiB
Commit graph

364 commits

This branch
This branch
All branches
Author SHA1 Message Date
Kresten Krab Thorup
8796575053 Fix dumb errors 2012-04-23 13:49:17 +02:00
Kresten Krab Thorup
a1c8bb40bd Send step_done when merge ends also 
… just to be nice, we'd get the {'DOWN', ...}
anyway
 2012-04-23 04:55:10 +02:00
Kresten Krab Thorup
25b4099eec Only compress if compressed size is smaller 2012-04-23 04:23:52 +02:00
Kresten Krab Thorup
d37b227936 Implement compression + block size 
option {compression, none|gzip|snappy}

... except right now using snappy is broken,
it seems that it causes bloom filters to
crash. Needs investigation.

option {block_size, 32768} 

... writes data to disk in chunks of ~32k.
 2012-04-23 03:49:08 +02:00
Kresten Krab Thorup
14ef03e06a Initial options 
Now we're ready to handle some options
 2012-04-23 02:20:47 +02:00
Kresten Krab Thorup
0718d33d7a Add {sync_strategy, sync | {seconds, N} | none} 
We should add o_sync also like bitcask
 2012-04-23 02:20:12 +02:00
Kresten Krab Thorup
3b451d5863 Fix folding 
The "blocking range fold" only works for modest
data sets, otherwise it gets prohibitively slow,
so for now we always do "snapshot range fold".
 2012-04-23 02:10:18 +02:00
Kresten Krab Thorup
ca98b124ff Merge remote-tracking branch 'refs/remotes/origin/master' 2012-04-22 23:55:07 +02:00
Kresten Krab Thorup
8f5f62cf70 Switch order ABCX -> CBAX in visualizer 
This makes it so that the youngest file is
always more left, and older files are always
more right.
 2012-04-22 23:53:31 +02:00
Kresten Krab Thorup
99fb1bee74 When opening a level, enforce just enough merge 
When re-opening a Hanoi data store, we need to
reestablish the invariant that there is always
room to inject a data file at the top level.

In a worst case scenario, every level has all of
A, B, and C; and thus needs to merge A+B -> X
fully in order to accommodate what the parent 
will inject. 2*BTREE_SIZE(Level) >= sizeof(A+B)
 2012-04-22 23:49:39 +02:00
Kresten Krab Thorup
8694cc118f Specify infinity as gen_server:call timeout 2012-04-22 23:30:25 +02:00
Gregory Burd
67315ca796 Implementation notes belong in "DESIGN.md", not "README.md". 2012-04-22 09:27:17 -04:00
Kresten Krab Thorup
d17e448615 new visualizer 2012-04-22 15:14:14 +02:00
Gregory Burd
85b09530ee Update todo items. 2012-04-21 17:24:37 -04:00
Gregory Burd
638f2d56ee Merge pull request #3 from basho/rename-to-hanoi 
Rename "lsm_btree" to "hanoi".
 2012-04-21 12:30:07 -07:00
Gregory Burd
43f095b3f0 Rename "lsm-btree" to "hanoi". 2012-04-21 15:20:39 -04:00
Gregory Burd
fec14a1c51 Put the copyright/license header into a few overlooked files. 2012-04-19 18:09:01 -04:00
Gregory Burd
bd3e638a96 Merge branch 'master' into gsb-merge-krab-20120419 2012-04-19 18:03:19 -04:00
Gregory Burd
197914939c Removed redundant function. 2012-04-19 18:00:06 -04:00
Gregory Burd
23f6d76a72 Merge branch 'krab-incremental-merge' into gsb-merge-krab-20120419 
Conflicts:
	src/lsm_btree.erl
	src/lsm_btree.hrl
 2012-04-19 16:54:23 -04:00
Kresten Krab Thorup
6289602045 Improve incremental merge 
This change makes incremental merge be concurrent 
with filling up the nursery. So in stead of waiting
for an incremental merge to complete before returning
from insert, it

- blocks waiting for a possible previous incremental merge to complete
- issues a new incremental merge.

This improves put latencies, but not throughput.
 2012-04-19 22:33:27 +02:00
Kresten Krab Thorup
ee90944c62 Implement incremental insert 
This slows down insert to be log2(N), where N is
the total number of objects in the store.  The upside
is that it also removes the terrible worst case
scenarios for insert.
 2012-04-19 19:57:39 +02:00
Kresten Krab Thorup
3d80b164d5 Introduce 3rd file in each level to reduce worst-case 
Now, each level is comprised of 3 files,

   A=Oldest, B=Older, C=Old

As in [Overmars and Leeuwen, 1983]. As soon as we have A & B,
we initiate a merge, (to the M=New) file, i.e. we merge more
eagerly than previously.

Next step in this refactoring is to add a scheduler that enforces
some merge activity as part of a PUT.
 2012-04-19 16:07:11 +02:00
Gregory Burd
3e85a266a8 More things to do... 2012-04-18 17:40:21 -04:00
Gregory Burd
b14887f9da Merge branch 'master' of github.com:basho/lsm_btree 2012-04-18 17:12:50 -04:00
Gregory Burd
1da9a858d1 Keep track of todo items here 2012-04-18 17:12:40 -04:00
Gregory Burd
0a72ca1df9 The Log-Structured Merge-Tree (LSM-Tree) by Patrick O'Neil, Edward Cheng Dieter Gawlick, Elizabeth O'Neil 2012-04-18 17:06:24 -04:00
Gregory Burd
3f02eadc27 Too large a nursery opens up the potential for long (in seconds) merges. 2012-04-18 16:49:57 -04:00
Kresten Krab Thorup
4e53b0a083 Allow fold worker to send {fold_results, PID, KVs} 
Not just individual KVs, but lists of KVs
 2012-04-18 09:28:59 +02:00
Kresten Krab Thorup
b70d2af1da Add doc to lsm_btree.hrl 2012-04-18 09:26:54 +02:00
Kresten Krab Thorup
4b91c71781 Invoke local rebar 
Some of us doesn't have rebar on the general path
 2012-04-18 09:26:21 +02:00
Kresten Krab Thorup
c3f916c350 Make fold_worker link to the consumer 
This should ensure proper cleanup if a process
calling fold exits while folding.
 2012-04-18 09:25:47 +02:00
Kresten Krab Thorup
5facc3df18 Undo fancy-pancy sext key encoding 
Sadly, this didn't work
Have to investigate more…
 2012-04-16 21:51:01 -04:00
Kresten Krab Thorup
1cf4805da5 Utilize sext to optimize bucket range queries 
Also re-fix the range arguments which were broken
in a previous commit.
 2012-04-16 21:51:01 -04:00
Kresten Krab Thorup
e7a621e449 Handle exceptions inside sync_fold_range 
Someone tried (tsk, tsk) to terminate a fold operation
by throwing an exception.  Now we also should handle
such situations gracefully.
 2012-04-16 21:49:15 -04:00
Steve Vinoski
4dab3a65e5 remove throw statements from fold functions 2012-04-16 21:42:04 -04:00
Kresten Krab Thorup
454a111ad7 Handle exceptions inside sync_fold_range 
Someone tried (tsk, tsk) to terminate a fold operation
by throwing an exception.  Now we also should handle
such situations gracefully.
 2012-04-17 00:29:28 +02:00
Steve Vinoski
a911734134 lsm_btree:sync_fold_range/5 no longer exists, use /4 instead 2012-04-16 17:46:43 -04:00
Steve Vinoski
79872680da rename temp_riak_kv_backend to lsm_btree_temp_riak_kv_backend 
The module temp_riak_kv_backend is already used in another experimental
storage backend. Rename it to avoid collisions when trying to use both
backends in riak together.
 2012-04-16 17:25:51 -04:00
Steve Vinoski
eefada16ac restore lookup/2 function, for compatibility 2012-04-16 16:50:55 -04:00
Kresten Krab Thorup
5f9f5c18d6 Add #btree_range to all fold ops 
lets you specify this for fold operations

  #btree_range {
     from_key :: binary(),
     from_inclusive = true :: boolean(),
     to_key :: binary() | undefined,
     to_inclusive = false :: boolean(),
     limit = undefined :: pos_integer() | undefined
  }
 2012-04-16 16:46:57 -04:00
Kresten Krab Thorup
9a7959ff4c Simplify KEY_IN_RANGE macro 2012-04-16 16:45:25 -04:00
Gregory Burd
8409df447d Another notfound -> not_found fix 2012-04-16 07:31:14 -04:00
Gregory Burd
5c5934549f The nursery was far too small, increase it from 32 to 8192 objects. 2012-04-15 17:23:42 -04:00
Gregory Burd
e81a3480ab Fix silly oversight with variable names, add better data dir prep. 2012-04-15 16:56:19 -04:00
Gregory Burd
4eaa02ac3f Formatting 2012-04-15 16:55:12 -04:00
Gregory Burd
b4823d3e8f Formatting 2012-04-15 16:54:27 -04:00
Gregory Burd
61d360550e Use not_found uniformly (rather than notfound). 2012-04-15 15:34:42 -04:00
Gregory Burd
5338a07c54 Don't prefix vnode directories with the backend name 2012-04-15 14:36:05 -04:00
Gregory Burd
a04475b871 Add lsm_btree into the list of applications in riak_kv.app.src 2012-04-15 14:35:31 -04:00