machi/prototype/tango/README.md

# Tango prototype

This is a quick hack, just to see how quick & easy it might be to
build Tango on top of corfurl.  It turned out to be pretty quick and
easy.

This prototype does not include any datatype-specific APIs, such as an
HTTP REST interface for manipulating a queue.  The current API is
native Erlang only.  However, because the Tango client communicates to
the underlying CORFU log via the `corfurl` interface, this
implementation is powerful enough to run concurrently on multiple
Erlang nodes.

This implementation does not follow the same structure as described in
the Tango paper.  I made some changes, based on some guesses/partial
understanding of the paper.  If I were to start over again, I'd try to
use the exact same naming scheme & structure suggested by the paper.

## Testing environment

Tested using Erlang/OTP R16B and Erlang/OTP 17, both on OS X.

It ought to "just work" on other versions of Erlang and on other OS
platforms, but sorry, I haven't tested it.

Use `make` and `make test` to compile and run unit tests.
Note that the Makefile assumes that the rebar utility is available
somewhere in your path.

## Data types implemented

* OID mapper
* Simple single-value register
* Map (i.e., multi-value register or basic key-value store)
* Queue
    * Used the Erlang/OTP `queue.erl` library for rough inspiration
    * Operations: is_empty, length, peek, to_list, member, in, out,
      reverse, filter.
    * Queue mutation operations are not idempotent with respect to
      multiple writes in the underlying CORFU log, e.g., due to CORFU
      log reconfiguration or partial write error/timeouts.

## Experimental idea: built-in OID checkpointing

I was toying with the idea of adding a Tango "history splicing"
operation that could make the implementation per-OID checkpoint &
garbage collection (and CORFU-level trimming) operations much easier.
I think that this might be a very good idea and that it deserves more
research & work.

The implementation of the checkpointing & splicing as it is today is
flawed.  See the TODO list below for more details.

## Information about the Tango paper

"Tango: Distributed Data Structures over a Shared Log"

Balakrishnan, Malkhi, Wobber, Wu, Brabhakaran, Wei, Davis, Rao, Zou, Zuck

Describes a framework for developing data structures that reside
persistently within a CORFU log: the log *is* the database/data
structure store.

http://www.snookles.com/scottmp/corfu/Tango.pdf

See also, `../corfu/docs/corfurl.md` for more information on CORFU
research papers.

## TODO list

__ The src/corfu* files in this sub-repo differ from the original
   prototype source files in the ../corfu sub-repo, sorry!

__ The current checkpoint implementation is fundamentally broken and
   needs a rewrite, or else.
   This issue is not mentioned at all in the Tango paper.

   option 1: fix checkpoint to be 100% correct
   option 2: checkpointing is for the weak and the memory-constrained, so
             don't bother.  Instead, rip out the current checkpoint code,
             period. 
   option 3: other

xx Checkpoint fix option #1: history splicing within the same OID?

xx Checkpoint fix option #2: checkpoint to a new OID, history writes to both
                             OIDs during the CP, then a marker in the old OID
                             to switch over to the new OID?

History splicing has a flaw that I belive just won't work.  The switch to a
new OID has problems with updates written to the old OID before and before the
new checkpoint has finished.

I believe that a checkpoint where:

* all Tango writes, checkpoint and non-checkpoint alike, are noted with
  a checkpoint number.
* that checkpoint number is strictly increasing
* a new checkpoint has a new checkpoint number
* scans ignore blocks with checkpoint numbers larger than the current
  active checkpoint #, until the checkpoint is complete.

... ought to work correctly.