d37b227936
option {compression, none|gzip|snappy}
... except right now using snappy is broken,
it seems that it causes bloom filters to
crash. Needs investigation.
option {block_size, 32768}
... writes data to disk in chunks of ~32k.
52 lines
3.2 KiB
Text
52 lines
3.2 KiB
Text
* hanoi (in order of priority)
|
|
* [2i] secondary index support
|
|
* atomic multi-commit/recovery
|
|
* add checkpoint/1 and sync/1 - flush pending writes to stable storage
|
|
(nursery:finish() and finish/flush any merges)
|
|
* [config] add config parameters on open
|
|
* {cache, bytes(), name} share max(bytes) cache named 'name' via etc
|
|
* [stats] statistics
|
|
* For each level {#merges, {merge-time-min, max, average}}
|
|
* [expiry] support for time based expiry, merge should eliminate expired data
|
|
* add @doc strings and and -spec's
|
|
* check to make sure every error returns with a reason {error, Reason}
|
|
* lager; check for uses of lager:error/2
|
|
* add version 1, crc to the files
|
|
* add compression via snappy (https://github.com/fdmanana/snappy-erlang-nif)
|
|
* add encryption
|
|
* adaptive nursery sizing
|
|
* add truncate/1 - quickly truncates a database to 0 items
|
|
* count/1 - return number of items currently in tree
|
|
* backpressure on fold operations
|
|
- The "sync_fold" creates a snapshot (hard link to btree files), which
|
|
provides consistent behavior but may use a lot of disk space if there is
|
|
a lot of insertion going on.
|
|
- The "async_fold" folds a limited number, and remembers the last key
|
|
serviced, then picks up from there again. So you could see intermittent
|
|
puts in a subsequent batch of results.
|
|
|
|
|
|
PHASE 2:
|
|
* hanoi
|
|
* Define a standard struct which is the metadata added at the end of the
|
|
file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written
|
|
in hanoi_writer:flush_nodes, and read in hanoi_reader:open2.
|
|
* [feature] compression, encryption on disk
|
|
|
|
|
|
|
|
REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS:
|
|
* nessdb https://code.google.com/p/nessdb/source/browse/LSM-BTREE?r=3a1df166a19505a2369dd954e8fc6d0a545f3d7b
|
|
* http://tokutek.com/downloads/mysqluc-2010-fractal-trees.pdf page 14+
|
|
* http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.44.2782&rep=rep1&type=pdf
|
|
|
|
|
|
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in hanoi.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
|
|
|
|
2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads.
|
|
|
|
3: Also, there is no caching of read nodes. So every time a btree node is visited it is also read from disk and term_to_binary'ed. But we need a caching system for that to work well (https://github.com/cliffmoon/cherly is difficult to build), it needs to be rebar-ified.
|
|
|
|
4: Also, the format for btree nodes could probably be optimized. Right now it's just binary_to_term of a key/value list as far as I remember. Perhaps we dont have to deserialize the entire thing.
|
|
|
|
5: It might also be good to employ a scheduler (github.com/esl/jobs<http://github.com/esl/jobs>) for issuing merges; because I think that it can be a problem for the OS if there are too many merges going on at the same time.
|