hanoidb/TODO

46 lines
2.8 KiB
Text
Raw Permalink Normal View History

2012-04-24 14:37:45 +00:00
* Phase 1: Minimum viable product (in order of priority)
* configurable TOP_LEVEL size
* test new snappy compression support
2012-04-25 18:07:06 +00:00
* status and statistics
2012-04-24 14:37:45 +00:00
* for each level {#merges, {merge-time-min, max, average}}
2012-04-21 21:24:37 +00:00
* add @doc strings and and -spec's
* check to make sure every error returns with a reason {error, Reason}
2012-04-24 14:37:45 +00:00
2012-04-28 16:42:04 +00:00
2012-04-24 14:37:45 +00:00
* Phase 2: Production Ready
* dual-nursery
* cache for read-path
* {cache, bytes(), name} share max(bytes) cache named 'name' via etc
2012-05-07 12:57:58 +00:00
* snapshot entire database (fresh directory w/ hard links to all files)
* persist merge progress (to speed up re-opening a HanoiDB)
2012-05-11 10:30:20 +00:00
* support for future file format changes
* Define a standard struct which is the metadata added at the end of the
file, e.g. [btree-nodes] [meta-data] [offset of meta-data]. This is written
in hanoi_writer:flush_nodes, and read in hanoi_reader:open2.
2012-04-24 14:37:45 +00:00
* Phase 3: Wish List
2012-04-21 21:24:37 +00:00
* add truncate/1 - quickly truncates a database to 0 items
* count/1 - return number of items currently in tree
2012-04-24 14:37:45 +00:00
* adaptive nursery sizing
2012-04-21 21:24:37 +00:00
* backpressure on fold operations
2012-04-18 21:12:40 +00:00
- The "sync_fold" creates a snapshot (hard link to btree files), which
provides consistent behavior but may use a lot of disk space if there is
a lot of insertion going on.
- The "async_fold" folds a limited number, and remembers the last key
serviced, then picks up from there again. So you could see intermittent
puts in a subsequent batch of results.
2012-04-24 14:37:45 +00:00
* add block-level encryption support
2012-04-18 21:12:40 +00:00
2012-04-24 14:37:45 +00:00
## NOTES:
2012-04-18 21:12:40 +00:00
1: make the "first level" have more thatn 2^5 entries (controlled by the constant TOP_LEVEL in hanoi.hrl); this means a new set of files is opened/closed/merged for every 32 insert/updates/deletes. Setting this higher will just make the nursery correspondingly larger, which should be absolutely fine.
2012-04-18 21:12:40 +00:00
2: Right now, the streaming btree writer emits a btree page based on number of elements. This could be changed to be based on the size of the node (say, some block-size boudary) and then add padding at the end so that each node read becomes a clean block transfer. Right now, we're probably taking way to many reads.
3: Also, there is no caching of read nodes. So every time a btree node is visited it is also read from disk and term_to_binary'ed. But we need a caching system for that to work well (https://github.com/cliffmoon/cherly is difficult to build), it needs to be rebar-ified.
4: Also, the format for btree nodes could probably be optimized. Right now it's just binary_to_term of a key/value list as far as I remember. Perhaps we dont have to deserialize the entire thing.
5: It might also be good to employ a scheduler (github.com/esl/jobs<http://github.com/esl/jobs>) for issuing merges; because I think that it can be a problem for the OS if there are too many merges going on at the same time.