Institutionalize the way hanoi_level handles RPC.
This is embodied in a new module, which should be
pushed to plain_fsm, but we'll keep it here for
now.
Now incremental merge has a new strategy.
In stead of doing the same amount of merge
work at all levels, we now compute the total
merge work load, and do as much as possible
on the first level, subtract work done, and
delegate to the next level, etc.
The effect of this is that we do more IO on
fewer files, improving sequential-ness of
the workload involved in the incremental merge.
This refactoring just adds the stat to the
master gen_server of a Hanoi instance to
know the current number of levels. Until now,
we've only held a reference to the current
top level.
If we're opening a hanoi store configured with
smaller nursery size than the default, then
we need to make sure that we also open the
small levels.
Future feature is to actually squash the
smaller levels.
This improves recovery two-fold:
1. make sure that we actually wait for initial
merge to complete (issue incremental_merge(0))
2. compute minimum required merge work for merge
to establish invariant that there's room
for a new nursery inject any time.
option {compression, none|gzip|snappy}
... except right now using snappy is broken,
it seems that it causes bloom filters to
crash. Needs investigation.
option {block_size, 32768}
... writes data to disk in chunks of ~32k.
When re-opening a Hanoi data store, we need to
reestablish the invariant that there is always
room to inject a data file at the top level.
In a worst case scenario, every level has all of
A, B, and C; and thus needs to merge A+B -> X
fully in order to accommodate what the parent
will inject. 2*BTREE_SIZE(Level) >= sizeof(A+B)
This change makes incremental merge be concurrent
with filling up the nursery. So in stead of waiting
for an incremental merge to complete before returning
from insert, it
- blocks waiting for a possible previous incremental merge to complete
- issues a new incremental merge.
This improves put latencies, but not throughput.
This slows down insert to be log2(N), where N is
the total number of objects in the store. The upside
is that it also removes the terrible worst case
scenarios for insert.
Now, each level is comprised of 3 files,
A=Oldest, B=Older, C=Old
As in [Overmars and Leeuwen, 1983]. As soon as we have A & B,
we initiate a merge, (to the M=New) file, i.e. we merge more
eagerly than previously.
Next step in this refactoring is to add a scheduler that enforces
some merge activity as part of a PUT.
