greg/hanoidb
1
0
Fork 0
Code Issues 10 Pull requests 3 Projects Releases Packages Wiki Activity

Update todo items.

Browse source
This commit is contained in:
Gregory Burd 2012-04-21 17:24:37 -04:00
parent 638f2d56ee
commit 85b09530ee
1 changed files with 22 additions and 36 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

58
TODO
View file

@ -1,21 +1,24 @@
* hanoi * lsm_btree
* [cleanup] add @doc strings and and -spec's * [2i] secondary index support
* [cleanup] check to make sure every error returns with a reason {error, Reason} * atomic multi-commit/recovery
* [feature] statistics * add checkpoint/1 and sync/1 - flush pending writes to stable storage
* [feature] use lager for error messages (nursery:finish() and finish/flush any merges)
* [enhancement] add crc or something to the files * [config] add config parameters on open
* [feature] add config parameters on open
* {sync, boolean()} fdsync or not on write * {sync, boolean()} fdsync or not on write
* {cache, bytes(), name} share max(bytes) cache named 'name' via etc * {cache, bytes(), name} share max(bytes) cache named 'name' via etc
* [enhancement] use etc/emmap to access/cache files * [stats] statistics
* [enhancement] adaptive nursery sizing * For each level {#merges, {merge-time-min, max, average}}
* [feature] support for time based expiry, merge should eliminate expired data * [expiry] support for time based expiry, merge should eliminate expired data
* [feature] add truncate/1 - quickly truncates a database to 0 items * add @doc strings and and -spec's
* [feature] add sync/1 - flush pending writes to disk (aka checkpoint) * check to make sure every error returns with a reason {error, Reason}
(nursery:finish() and finish/flush any merges) * lager; check for uses of lager:error/2
* [feature] count/1 - return number of items currently in tree * add version 1, crc to the files
* [feature] "group" commit - ability to make many k/v add/update/deletes atomic (for 2i) * add compression via snappy (https://github.com/fdmanana/snappy-erlang-nif)
* [enhancement] backpressure on fold operations * add encryption
* adaptive nursery sizing
* add truncate/1 - quickly truncates a database to 0 items
* count/1 - return number of items currently in tree
* backpressure on fold operations
- The "sync_fold" creates a snapshot (hard link to btree files), which - The "sync_fold" creates a snapshot (hard link to btree files), which
provides consistent behavior but may use a lot of disk space if there is provides consistent behavior but may use a lot of disk space if there is
a lot of insertion going on. a lot of insertion going on.
@ -23,31 +26,14 @@
serviced, then picks up from there again. So you could see intermittent serviced, then picks up from there again. So you could see intermittent
puts in a subsequent batch of results. puts in a subsequent batch of results.
* riak_kv_hanoie_backend
* add support for time-based expiry
* finish support for 2i
* add stats collection
- For each level {#merges, {merge-time-min, max, average}}
PHASE 2: PHASE 2:
* hanoi * lsm_btree
* Define a standard struct which is the metadata added at the end of the * Define a standard struct which is the metadata added at the end of the
file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written
in hanoi_writer:flush_nodes, and read in hanoi_reader:open2. in lsm_btree_writer:flush_nodes, and read in lsm_btree_reader:open2.
* [feature] compression, encryption on disk * [feature] compression, encryption on disk
PHASE 3:
* lsm_ixdb
* hanoi{btree, trie, ...} support for sub-databases and associations with
different index types
* [major change] add more CAPABILITIES such as
test-and-set(Fun, Key, Value) - to compare a vclock quickly, to speed up
the get/put patch for every update
* [enhancement] change encoding/layout of data on disk using sub-databases
and secondary indexes
bucket/key{meta[], data} -> ??
REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS: REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS:
@ -56,7 +42,7 @@ REVIEW LITERATURE AND OTHER SIMILAR IMPLEMENTATAIONS:
* http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.44.2782&rep=rep1&type=pdf * http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.44.2782&rep=rep1&type=pdf
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in hanoi.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine. 1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in lsm_btree.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads. 2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads.