machi/doc/src.high-level/high-level-chain-mgr.tex

%% \documentclass[]{report}
\documentclass[preprint,10pt]{sigplanconf}
% The following \documentclass options may be useful:

% preprint      Remove this option only once the paper is in final form.
% 10pt          To set in 10-point type instead of 9-point.
% 11pt          To set in 11-point type instead of 9-point.
% authoryear    To obtain author/year citation style instead of numeric.

% \usepackage[a4paper]{geometry}
\usepackage[dvips]{graphicx}            % to include images
%\usepackage{pslatex}	    % to use PostScript fonts

\begin{document}

%%\special{papersize=8.5in,11in}
%%\setlength{\pdfpageheight}{\paperheight}
%%\setlength{\pdfpagewidth}{\paperwidth}

\conferenceinfo{}{} 
\copyrightyear{2014} 
\copyrightdata{978-1-nnnn-nnnn-n/yy/mm} 
\doi{nnnnnnn.nnnnnnn}

\titlebanner{Draft \#0, April 2014}
\preprintfooter{Draft \#0, April 2014}

\title{Machi Chain Replication: management theory and design}
\subtitle{Includes ``humming consensus'' overview}

\authorinfo{Basho Japan KK}{}

\maketitle

\section{Origins}
\label{sec:origins}

This document was first written during the autumn of 2014 for a
Basho-only internal audience.  Since its original drafts, Machi has
been designated by Basho as a full open source software project.  This
document has been rewritten in 2015 to address an external audience.
For an overview of the design of the larger Machi system, please see
\cite{machi-design}.

\section{Abstract}
\label{sec:abstract}

We describe the self-management and self-reliance
goals of the algorithm: preserve data integrity, advance the current
state of the art, and supporting multiple consisistency levels.

TODO Fix, after all of the recent changes to this document.

A discussion of ``humming consensus'' follows next.  This type of
consensus does not require active participation by all or even a
majority of participants to make decisions.  Machi's chain manager
bases its logic on humming consensus to make decisions about how to
react to changes in its environment, e.g. server crashes, network
partitions, and changes by Machi cluster admnistrators.  Once a
decision is made during a virtual time epoch, humming consensus will
eventually discover if other participants have made a different
decision during that epoch.  When a differing decision is discovered,
new time epochs are proposed in which a new consensus is reached and
disseminated to all available participants.

\section{Introduction}
\label{sec:introduction}

\subsection{What does "self-management" mean?}
\label{sub:self-management}

For the purposes of this document, chain replication self-management
is the ability for the $N$ nodes in an $N$-length chain replication chain
to manage the state of the chain without requiring an external party
to participate.  Chain state includes:

\begin{itemize}
\item Preserve data integrity of all data stored within the chain.  Data
   loss is not an option.
\item Stably preserve knowledge of chain membership (i.e. all nodes in
   the chain, regardless of operational status). A systems
   administrators is expected to make "permanent" decisions about
   chain membership.
\item Use passive and/or active techniques to track operational
   state/status, e.g., up, down, restarting, full data sync, partial
   data sync, etc.
\item Choose the run-time replica ordering/state of the chain, based on
   current member status and past operational history.  All chain
   state transitions must be done safely and without data loss or
   corruption.
\item As a new node is added to the chain administratively or old node is
   restarted, add the node to the chain safely and perform any data
   synchronization/"repair" required to bring the node's data into
   full synchronization with the other nodes.
\end{itemize}

\subsection{Ultimate goal: Preserve data integrity of Chain Replicated data}

Preservation of data integrity is paramount to any chain state
management technique for Machi.  Even when operating in an eventually
consistent mode, Machi must not lose data without cause outside of all
design, e.g., all particpants crash permanently.

\subsection{Goal: better than state-of-the-art Chain Replication management}

We hope/believe that this new self-management algorithem can improve
the current state-of-the-art by eliminating all external management
entities.  Current state-of-the-art for management of chain
replication chains is discussed below, to provide historical context.

\subsubsection{``Leveraging Sharding in the Design of Scalable Replication Protocols'' by Abu-Libdeh, van Renesse, and Vigfusson}
\label{ssec:elastic-replication}
Multiple chains are arranged in a ring (called a "band" in the paper).
The responsibility for managing the chain at position N is delegated
to chain N-1.  As long as at least one chain is running, that is
sufficient to start/bootstrap the next chain, and so on until all
chains are running.  The paper then estimates mean-time-to-failure
(MTTF) and suggests a "band of bands" topology to handle very large
clusters while maintaining an MTTF that is as good or better than
other management techniques.

{\bf NOTE:} If the chain self-management method proposed for Machi does not
succeed, this paper's technique is our best fallback recommendation.

\subsubsection{An external management oracle, implemented by ZooKeeper}
\label{ssec:an-oracle}
This is not a recommendation for Machi: we wish to avoid using ZooKeeper.
However, many other open source software products use
ZooKeeper for exactly this kind of data replica management problem.

\subsubsection{An external management oracle, implemented by Riak Ensemble}

This is a much more palatable choice than option~\ref{ssec:an-oracle}
above.  We also
wish to avoid an external dependency on something as big as Riak
Ensemble.  However, if it comes between choosing Riak Ensemble or
choosing ZooKeeper, the choice feels quite clear: Riak Ensemble will
win, unless there is some critical feature missing from Riak
Ensemble.  If such an unforseen missing feature is discovered, it
would probably be preferable to add the feature to Riak Ensemble
rather than to use ZooKeeper (and document it and provide product
support for it and so on...).

\subsection{Goal: Support both eventually consistent \& strongly consistent modes of operation}

Machi's first use case is for Riak CS, as an eventually consistent
store for CS's "block" storage.  Today, Riak KV is used for "block"
storage.  Riak KV is an AP-style key-value store; using Machi in an
AP-style mode would match CS's current behavior from points of view of
both code/execution and human administrator exectations.

Later, we wish the option of using CP support to replace other data
store services that Riak KV provides today.  (Scope and timing of such
replacement TBD.)

We believe this algorithm allows a Machi cluster to fragment into
arbitrary islands of network partition, all the way down to 100\% of
members running in complete network isolation from each other.
Furthermore, it provides enough agreement to allow
formerly-partitioned members to coordinate the reintegration \&
reconciliation of their data when partitions are healed.

\subsection{Anti-goal: minimize churn}

This algorithm's focus is data safety and not availability.  If
participants have differing notions of time, e.g., running on
extremely fast or extremely slow hardware, then this algorithm will
"churn" in different states where the chain's data would be
effectively unavailable.

In practice, however, any series of network partition changes that
case this algorithm to churn will cause other management techniques
(such as an external "oracle") similar problems.
{\bf [Proof by handwaving assertion.]}
See also: Section~\ref{sub:time-model}

\section{Assumptions}
\label{sec:assumptions}

Given a long history of consensus algorithms (viewstamped replication,
Paxos, Raft, et al.), why bother with a slightly different set of
assumptions and a slightly different protocol?

The answer lies in one of our explicit goals: to have an option of
running in an "eventually consistent" manner.  We wish to be able to
make progress, i.e., remain available in the CAP sense, even if we are
partitioned down to a single isolated node.  VR, Paxos, and Raft
alone are not sufficient to coordinate service availability at such
small scale.

\subsection{The CORFU protocol is correct}

This work relies tremendously on the correctness of the CORFU
protocol \cite{corfu1}, a cousin of the Paxos protocol.
If the implementation of
this self-management protocol breaks an assumption or prerequisite of
CORFU, then we expect that Machi's implementation will be flawed.

\subsection{Communication model: asyncronous message passing}

The network is unreliable: messages may be arbitrarily dropped and/or
reordered.  Network partitions may occur at any time.
Network partitions may be asymmetric, e.g., a message can be sent
from $A \rightarrow B$, but messages from $B \rightarrow A$ can be
lost, dropped, and/or arbitrarily delayed.

System particpants may be buggy but not actively malicious/Byzantine.

\subsection{Time model}
\label{sub:time-model}

Our time model is per-node wall-clock time clocks, loosely
synchronized by NTP.

The protocol and algorithm presented here do not specify or require any
timestamps, physical or logical.  Any mention of time inside of data
structures are for human/historic/diagnostic purposes only.

Having said that, some notion of physical time is suggested for
purposes of efficiency.  It's recommended that there be some "sleep
time" between iterations of the algorithm: there is no need to "busy
wait" by executing the algorithm as quickly as possible.  See below,
"sleep intervals between executions".

\subsection{Failure detector model: weak, fallible, boolean}

We assume that the failure detector that the algorithm uses is weak,
it's fallible, and it informs the algorithm in boolean status
updates/toggles as a node becomes available or not.

If the failure detector is fallible and tells us a mistaken status
change, then the algorithm will "churn" the operational state of the
chain, e.g. by removing the failed node from the chain or adding a
(re)started node (that may not be alive) to the end of the chain.
Such extra churn is regrettable and will cause periods of delay as the
"rough consensus" (decribed below) decision is made.  However, the
churn cannot (we assert/believe) cause data loss.

\subsection{Use of the ``wedge state''}

A participant in Chain Replication will enter "wedge state", as
described by the Machi high level design \cite{machi-design} and by CORFU,
when it receives information that
a newer projection (i.e., run-time chain state reconfiguration) is
available.  The new projection may be created by a system
administrator or calculated by the self-management algorithm.
Notification may arrive via the projection store API or via the file
I/O API.

When in wedge state, the server will refuse all file write I/O API
requests until the self-management algorithm has determined that
"rough consensus" has been decided (see next bullet item).  The server
may also refuse file read I/O API requests, depending on its CP/AP
operation mode.

\subsection{Use of ``humming consensus''}

CS literature uses the word "consensus" in the context of the problem
description at
{\tt http://en.wikipedia.org/wiki/ Consensus\_(computer\_science)\#Problem\_description}.
This traditional definition differs from what is described here as
``humming consensus''.

"Humming consensus" describes
consensus that is derived only from data that is visible/known at the current
time.  This implies that a network partition may be in effect and that
not all chain members are reachable.  The algorithm will calculate
an approximate consensus despite not having input from all/majority
of chain members.  Humming consensus may proceed to make a
decision based on data from only a single participant, i.e., only the local
node.

See Section~\ref{sec:humming-consensus} for detailed discussion.

\section{The projection store}

The Machi chain manager relies heavily on a key-value store of
write-once registers called the ``projection store''.
Each participating chain node has its own projection store.
The store's key is a positive integer;
the integer represents the epoch number of the projection.  The
store's value is either the special `unwritten' value\footnote{We use
  $\emptyset$ to denote the unwritten value.} or else an
application-specific binary blob that is immutable thereafter.

The projection store is vital for the correct implementation of humming
consensus (Section~\ref{sec:humming-consensus}).

All parts store described below may be read by any cluster member.

\subsection{The publicly-writable half of the projection store}

The publicly-writable projection store is used to share information
during the first half of the self-management algorithm.  Any chain
member may write a projection to this store.

\subsection{The privately-writable half of the projection store}

The privately-writable projection store is used to store the humming consensus
result that has been chosen by the local node.  Only
the local server may write values into this store.

The private projection store serves multiple purposes, including:

\begin{itemize}
\item remove/clear the local server from ``wedge state''
\item act as the store of record for chain state transitions
\item communicate to remote nodes the past states and current operational
  state of the local node
\end{itemize}

\section{Projections: calculation, storage, and use}
\label{sec:projections}

Machi uses a ``projection'' to determine how its Chain Replication replicas
should operate; see \cite{machi-design} and
\cite{corfu1}.  At runtime, a cluster must be able to respond both to
administrative changes (e.g., substituting a failed server box with
replacement hardware) as well as local network conditions (e.g., is
there a network partition?).

The concept of a projection is borrowed
from CORFU but has a longer history, e.g., the Hibari key-value store
\cite{cr-theory-and-practice} and goes back in research for decades,
e.g., Porcupine \cite{porcupine}.

\subsection{The projection data structure}
\label{sub:the-projection}

{\bf NOTE:} This section is a duplicate of the ``The Projection and
the Projection Epoch Number'' section of \cite{machi-design}.

The projection data
structure defines the current administration \& operational/runtime
configuration of a Machi cluster's single Chain Replication chain.
Each projection is identified by a strictly increasing counter called
the Epoch Projection Number (or more simply ``the epoch'').

Projections are calculated by each server using input from local
measurement data, calculations by the server's chain manager
(see below), and input from the administration API.
Each time that the configuration changes (automatically or by
administrator's request), a new epoch number is assigned
to the entire configuration data structure and is distributed to
all servers via the server's administration API.  Each server maintains the
current projection epoch number as part of its soft state.

Pseudo-code for the projection's definition is shown in
Figure~\ref{fig:projection}.  To summarize the major components:

\begin{figure}
\begin{verbatim}
-type m_server_info() :: {Hostname, Port,...}.
-record(projection, {
            epoch_number    :: m_epoch_n(),
            epoch_csum      :: m_csum(),
            creation_time   :: now(),
            author_server   :: m_server(),
            all_members     :: [m_server()],
            active_upi      :: [m_server()],
            active_all      :: [m_server()],
            down_members    :: [m_server()],
            dbg_annotations :: proplist()
        }).
\end{verbatim}
\caption{Sketch of the projection data structure}
\label{fig:projection}
\end{figure}

\begin{itemize}
\item {\tt epoch\_number} and {\tt epoch\_csum} The epoch number and
  projection checksum are unique identifiers for this projection.
\item {\tt creation\_time} Wall-clock time, useful for humans and
  general debugging effort.
\item {\tt author\_server} Name of the server that calculated the projection.
\item {\tt all\_members} All servers in the chain, regardless of current
  operation status.  If all operating conditions are perfect, the
  chain should operate in the order specified here.
\item {\tt active\_upi} All active chain members that we know are
  fully repaired/in-sync with each other and therefore the Update
  Propagation Invariant (Section~\ref{sub:upi} is always true.
\item {\tt active\_all} All active chain members, including those that
  are under active repair procedures.
\item {\tt down\_members} All members that the {\tt author\_server}
  believes are currently down or partitioned.
\item {\tt dbg\_annotations} A ``kitchen sink'' proplist, for code to
  add any hints for why the projection change was made, delay/retry
  information, etc.
\end{itemize}

\subsection{Why the checksum field?}

According to the CORFU research papers, if a server node $S$ or client
node $C$ believes that epoch $E$ is the latest epoch, then any information
that $S$ or $C$ receives from any source that an epoch $E+\delta$ (where
$\delta > 0$) exists will push $S$ into the "wedge" state and $C$ into a mode
of searching for the projection definition for the newest epoch.

In the humming consensus description in
Section~\ref{sec:humming-consensus}, it should become clear that it's
possible to have a situation where two nodes make proposals
for a single epoch number.  In the simplest case, assume a chain of
nodes $A$ and $B$.  Assume that a symmetric network partition between
$A$ and $B$ happens. Also, let's assume that operating in
AP/eventually consistent mode.

On $A$'s network-partitioned island, $A$ can choose
an active chain definition of {\tt [A]}.
Similarly $B$ can choose a definition of {\tt [B]}.  Both $A$ and $B$
might choose the
epoch for their proposal to be \#42.  Because each are separated by
network partition, neither can realize the conflict.

When the network partition heals, it can become obvious to both
servers that there are conflicting values for epoch \#42.  If we
use CORFU's protocol design, which identifies the epoch identifier as
an integer only, then the integer 42 alone is not sufficient to
discern the differences between the two projections.

Humming consensus requires that any projection be identified by both
the epoch number and the projection checksum, as described in
Section~\ref{sub:the-projection}.

\section{Phases of projection change}
\label{sec:phases-of-projection-change}

Machi's use of projections is in four discrete phases and are
discussed below: network monitoring,
projection calculation, projection storage, and
adoption of new projections.  The phases are described in the
subsections below.  The reader should then be able to recognize each
of these phases when reading the humming consensus algorithm
description in Section~\ref{sec:humming-consensus}.

\subsection{Network monitoring}
\label{sub:network-monitoring}

Monitoring of local network conditions can be implemented in many
ways.  None are mandatory, as far as this RFC is concerned.
Easy-to-maintain code should be the primary driver for any
implementation.  Early versions of Machi may use some/all of the
following techniques:

\begin{itemize}
\item Internal ``no op'' FLU-level protocol request \& response.
\item Explicit connections of remote {\tt epmd} services, e.g., to
tell the difference between a dead Erlang VM and a dead
machine/hardware node.
\item Network tests via ICMP {\tt ECHO\_REQUEST}, a.k.a. {\tt ping(8)}
\end{itemize}

Output of the monitor should declare the up/down (or
available/unavailable) status of each server in the projection.  Such
Boolean status does not eliminate ``fuzzy logic'' or probabilistic
methods for determining status.  Instead, hard Boolean up/down status
decisions are required by the projection calculation phase
(Section~\ref{subsub:projection-calculation}).

\subsection{Projection data structure calculation}
\label{subsub:projection-calculation}

Each Machi server will have an independent agent/process that is
responsible for calculating new projections.  A new projection may be
required whenever an administrative change is requested or in response
to network conditions (e.g., network partitions).

Projection calculation will be a pure computation, based on input of:

\begin{enumerate}
\item The current projection epoch's data structure
\item Administrative request (if any)
\item Status of each server, as determined by network monitoring
(Section~\ref{sub:network-monitoring}).
\end{enumerate}

All decisions about {\em when} to calculate a projection must be made
using additional runtime information.  Administrative change requests
probably should happen immediately.  Change based on network status
changes may require retry logic and delay/sleep time intervals.

\subsection{Projection storage: writing}
\label{sub:proj-storage-writing}

Individual replicas of the projections written to participating
projection stores are not managed by Chain Replication --- if they
were, we would have a circular dependency!  See
Section~\ref{sub:proj-store-writing} for the technique for writing
projections to all participating servers' projection stores.

\subsection{Adoption of new projections}

The projection store's ``best value'' for the largest written epoch
number at the time of the read is projection used by the server.
If the read attempt for projection $P_p$
also yields other non-best values, then the
projection calculation subsystem is notified.  This notification
may/may not trigger a calculation of a new projection $P_{p+1}$ which
may eventually be stored and so
resolve $P_p$'s replicas' ambiguity.

\section{Humming consensus's management of multiple projection store}

Individual replicas of the projections written to participating
projection stores are not managed by Chain Replication.

An independent replica management technique very similar to the style
used by both Riak Core \cite{riak-core} and Dynamo is used.
The major difference is
that successful return status from (minimum) a quorum of participants
{\em is not required}.

\subsection{Read repair: repair only when unwritten}

The idea of ``read repair'' is also shared with Riak Core and Dynamo
systems.  However, there is a case read repair cannot truly ``fix'' a
key because two different values have been written by two different
replicas.

Machi's projection store is write-once, and there is no ``undo'' or
``delete'' or ``overwrite'' in the projection store API.  It doesn't
matter what caused the two different values.  In case of multiple
values, all participants in humming consensus merely agree that there
were multiple opinions at that epoch which must be resolved by the
creation and writing of newer projections with later epoch numbers.

Machi's projection store read repair can only repair values that are
unwritten, i.e., storing $\emptyset$.

\subsection{Projection storage: writing}
\label{sub:proj-store-writing}

All projection data structures are stored in the write-once Projection
Store that is run by each server.  (See also \cite{machi-design}.)

Writing the projection follows the two-step sequence below.

\begin{enumerate}
\item Write $P_{new}$ to the local projection store.  (As a side
  effect,
  this will trigger
  ``wedge'' status in the local server, which will then cascade to other
  projection-related behavior within that server.)
\item Write $P_{new}$ to the remote projection store of {\tt all\_members}.
  Some members may be unavailable, but that is OK.
\end{enumerate}

In cases of {\tt error\_written} status,
the process may be aborted and read repair
triggered.  The most common reason for {\tt error\_written} status
is that another actor in the system has
already calculated another (perhaps different) projection using the
same projection epoch number and that
read repair is necessary.  Note that {\tt error\_written} may also
indicate that another actor has performed read repair on the exact
projection value that the local actor is trying to write!

\section{Reading from the projection store}
\label{sec:proj-store-reading}

Reading data from the projection store is similar in principle to
reading from a Chain Replication-managed server system.  However, the
projection store does not use the strict replica ordering that
Chain Replication does.  For any projection store key $K_n$, the
participating servers may have different values for $K_n$.  As a
write-once store, it is impossible to mutate a replica of $K_n$.  If
replicas of $K_n$ differ, then other parts of the system (projection
calculation and storage) are responsible for reconciling the
differences by writing a later key,
$K_{n+x}$ when $x>0$, with a new projection.

Projection store reads are ``best effort''.  The projection used is chosen from
all replica servers that are available at the time of the read.  The
minimum number of replicas is only one: the local projection store
should always be available, even if no other remote replica projection
stores are available.

For any key $K$, different projection stores $S_a$ and $S_b$ may store
nothing (i.e., {\tt error\_unwritten} when queried) or store different
values, $P_a \ne P_b$, despite having the same projection epoch
number.  The following ranking rules are used to
determine the ``best value'' of a projection, where highest rank of
{\em any single projection} is considered the ``best value'':

\begin{enumerate}
\item An unwritten value is ranked at a value of $-1$.
\item A value whose {\tt author\_server} is at the $I^{th}$ position
  in the {\tt all\_members} list has a rank of $I$.
\item A value whose {\tt dbg\_annotations} and/or other fields have
  additional information may increase/decrease its rank, e.g.,
  increase the rank by $10.25$.
\end{enumerate}

Rank rules \#2 and \#3 are intended to avoid worst-case ``thrashing''
of different projection proposals.

The concept of ``read repair'' of an unwritten key is the same as
Chain Replication's.  If a read attempt for a key $K$ at some server
$S$ results in {\tt error\_unwritten}, then all of the other stores in
the {\tt \#projection.all\_members} list are consulted.  If there is a
unanimous value $V_{u}$ elsewhere, then $V_{u}$ is use to repair all
unwritten replicas.  If the value of $K$ is not unanimous, then the
``best value'' $V_{best}$ is used for the repair.  If all respond with
{\tt error\_unwritten}, repair is not required.

\section{Humming Consensus}
\label{sec:humming-consensus}

Sources for background information include:

\begin{itemize}
\item ``On Consensus and Humming in the IETF'' \cite{rfc-7282}, for
background on the use of humming during meetings of the IETF.

\item ``On `Humming Consensus', an allegory'' \cite{humming-consensus-allegory},
for an allegory in the style (?) of Leslie Lamport's original Paxos
paper.
\end{itemize}


\subsection{Summary of humming consensus}

"Humming consensus" describes
consensus that is derived only from data that is visible/known at the current
time.  This implies that a network partition may be in effect and that
not all chain members are reachable.  The algorithm will calculate
an approximate consensus despite not having input from all/majority
of chain members.  Humming consensus may proceed to make a
decision based on data from only a single participant, i.e., only the local
node.

\begin{itemize}

\item When operating in AP mode, i.e., in eventual consistency mode, humming
consensus may reconfigure chain of length $N$ into $N$
independent chains of length 1.  When a network partition heals, the
humming consensus is sufficient to manage the chain so that each
replica's data can be repaired/merged/reconciled safely.
Other features of the Machi system are designed to assist such
repair safely.

\item When operating in CP mode, i.e., in strong consistency mode, humming
consensus would require additional restrictions.  For example, any
chain that didn't have a minimum length of the quorum majority size of
all members would be invalid and therefore would not move itself out
of wedged state.  In very general terms, this requirement for a quorum
majority of surviving participants is also a requirement for Paxos,
Raft, and ZAB. See Section~\ref{sec:split-brain-management} for a
proposal to handle ``split brain'' scenarios while in CPU mode.

\end{itemize}

If a decision is made during epoch $E$, humming consensus will
eventually discover if other participants have made a different
decision during epoch $E$.  When a differing decision is discovered,
newer \& later time epochs are defined by creating new projections
with epochs numbered by $E+\delta$ (where $\delta > 0$).
The distribution of the $E+\delta$ projections will bring all visible
participants into the new epoch $E+delta$ and then into consensus.

\section{Just in case Humming Consensus doesn't work for us}

There are some unanswered questions about Machi's proposed chain
management technique.  The problems that we guess are likely/possible
include:

\begin{itemize}

\item Thrashing or oscillating between a pair (or more) of
  projections.  It's hoped that the ``best projection'' ranking system
  will be sufficient to prevent endless thrashing of projections, but
  it isn't yet clear that it will be.

\item Partial (and/or one-way) network splits which cause partially
  connected graphs of inter-node connectivity.  Groups of nodes that
  are completely isolated aren't a problem.  However, partially
  connected groups of nodes is an unknown.  Intuition says that
  communication (via the projection store) with ``bridge nodes'' in a
  partially-connected network ought to settle eventually on a
  projection with high rank, e.g., the projection on an island
  subcluster of nodes with the largest author node name.  Some corner
  case(s) may exist where this intuition is not correct.

\item CP Mode management via the method proposed in
  Section~\ref{sec:split-brain-management} may not be sufficient in
  all cases.

\end{itemize}

\subsection{Alternative: Elastic Replication}

Using Elastic Replication (Section~\ref{ssec:elastic-replication}) is
our preferred alternative, if Humming Consensus is not usable.

Elastic chain replication is a technique described in
\cite{elastic-chain-replication}.  It describes using multiple chains
to monitor each other, as arranged in a ring where a chain at position
$x$ is responsible for chain configuration and management of the chain
at position $x+1$.

\subsection{Alternative: Hibari's ``Admin Server''
  and Elastic Chain Replication}

See Section 7 of \cite{cr-theory-and-practice} for details of Hibari's
chain management agent, the ``Admin Server''.  In brief:

\begin{itemize}
\item The Admin Server is intentionally a single point of failure in
  the same way that the instance of Stanchion in a Riak CS cluster
  is an intentional single
  point of failure.  In both cases, strict
  serialization of state changes is more important than 100\%
  availability.

\item For higher availability, the Hibari Admin Server is usually
  configured in an active/standby manner.  Status monitoring and
  application failover logic is provided by the built-in capabilities
  of the Erlang/OTP application controller.

\end{itemize}

\section{``Split brain'' management in CP Mode}
\label{sec:split-brain-management}

Split brain management is a thorny problem.  The method presented here
is one based on pragmatics.  If it doesn't work, there isn't a serious
worry, because Machi's first serious use case all require only AP Mode.
If we end up falling back to ``use Riak Ensemble'' or ``use ZooKeeper'',
then perhaps that's
fine enough.  Meanwhile, let's explore how a
completely self-contained, no-external-dependencies
CP Mode Machi might work.

Wikipedia's description of the quorum consensus solution\footnote{See
  {\tt http://en.wikipedia.org/wiki/Split-brain\_(computing)}.} is nice
and short:

\begin{quotation}
A typical approach, as described by Coulouris et al.,[4] is to use a
quorum-consensus approach. This allows the sub-partition with a
majority of the votes to remain available, while the remaining
sub-partitions should fall down to an auto-fencing mode.
\end{quotation}

This is the same basic technique that
both Riak Ensemble and ZooKeeper use.  Machi's
extensive use of write-registers are a big advantage when implementing
this technique.  Also very useful is the Machi ``wedge'' mechanism,
which can automatically implement the ``auto-fencing'' that the
technique requires.  All Machi servers that can communicate with only
a minority of other servers will automatically ``wedge'' themselves
and refuse all requests for service until communication with the
majority can be re-established.

\subsection{The quorum: witness servers vs. full servers}

In any quorum-consensus system, at least $2f+1$ participants are
required to survive $f$ participant failures.  Machi can implement a
technique of ``witness servers'' servers to bring the total cost
somewhere in the middle, between $2f+1$ and $f+1$, depending on your
point of view.

A ``witness server'' is one that participates in the network protocol
but does not store or manage all of the state that a ``full server''
does.  A ``full server'' is a Machi server as
described by this RFC document.  A ``witness server'' is a server that
only participates in the projection store and projection epoch
transition protocol and a small subset of the file access API.
A witness server doesn't actually store any
Machi files.  A witness server is almost stateless, when compared to a
full Machi server.

A mixed cluster of witness and full servers must still contain at
least $2f+1$ participants.  However, only $f+1$ of them are full
participants, and the remaining $f$ participants are witnesses.  In
such a cluster, any majority quorum must have at least one full server
participant.

Witness FLUs are always placed at the front of the chain.  As stated
above, there may be at most $f$ witness FLUs.  A functioning quorum
majority
must have at least $f+1$ FLUs that can communicate and therefore
calculate and store a new unanimous projection.  Therefore, any FLU at
the tail of a functioning quorum majority chain must be full FLU.  Full FLUs
actually store Machi files, so they have no problem answering {\tt
  read\_req} API requests.\footnote{We hope that it is now clear that
  a witness FLU cannot answer any Machi file read API request.}

Any FLU that can only communicate with a minority of other FLUs will
find that none can calculate a new projection that includes a
majority of FLUs.  Any such FLU, when in CP mode, would then move to
wedge state and remain wedged until the network partition heals enough
to communicate with the majority side.  This is a nice property: we
automatically get ``fencing'' behavior.\footnote{Any FLU on the minority side
  is wedged and therefore refuses to serve because it is, so to speak,
  ``on the wrong side of the fence.''}

There is one case where ``fencing'' may not happen: if both the client
and the tail FLU are on the same minority side of a network partition.
Assume the client and FLU $F_z$ are on the "wrong side" of a network
split; both are using projection epoch $P_1$.  The tail of the
chain is $F_z$.

Also assume that the "right side" has reconfigured and is using
projection epoch $P_2$.  The right side has mutated key $K$.  Meanwhile,
nobody on the "right side" has noticed anything wrong and is happy to
continue using projection $P_1$.

\begin{itemize}
\item {\bf Option a}: Now the wrong side client reads $K$ using $P_1$ via
  $F_z$.  $F_z$ does not detect an epoch problem and thus returns an
  answer.  Given our assumptions, this value is stale.  For some
  client use cases, this kind of staleness may be OK in trade for
  fewer network messages per read \ldots so Machi may
  have a configurable option to permit it.
\item {\bf Option b}: The wrong side client must confirm that $P_1$ is
  in use by a full majority of chain members, including $F_z$.
\end{itemize}

Attempts using Option b will fail for one of two reasons.  First, if
the client can talk to a FLU that is using $P_2$, the client's
operation must be retried using $P_2$.  Second, the client will time
out talking to enough FLUs so that it fails to get a quorum's worth of
$P_1$ answers.  In either case, Option B will always fail a client
read and thus cannot return a stale value of $K$.

\subsection{Witness FLU data and protocol changes}

Some small changes to the projection's data structure
are required (relative to the initial spec described in
\cite{machi-design}).  The projection itself
needs new annotation to indicate the operating mode, AP mode or CP
mode.  The state type notifies the chain manager how to
react in network partitions and how to calculate new, safe projection
transitions and which file repair mode to use
(Section~\ref{sec:repair-entire-files}).
Also, we need to label member FLU servers as full- or
witness-type servers.

Write API requests are processed by witness servers in {\em almost but
  not quite} no-op fashion.  The only requirement of a witness server
is to return correct interpretations of local projection epoch
numbers, via the {\tt error\_bad\_epoch} and {\tt error\_wedged} error
codes.  In fact, a new API call is sufficient for querying witness
servers: {\tt \{check\_epoch, m\_epoch()\}}.
Any client write operation sends the {\tt
  check\_\-epoch} API command to witness FLUs and sends the usual {\tt
  write\_\-req} command to full FLUs.

\section{The safety of projection epoch transitions}
\label{sec:safety-of-transitions}

Machi uses the projection epoch transition algorithm and
implementation from CORFU, which is believed to be safe.  However,
CORFU assumes a single, external, strongly consistent projection
store.  Further, CORFU assumes that new projections are calculated by
an oracle that the rest of the CORFU system agrees is the sole agent
for creating new projections.  Such an assumption is impractical for
Machi's intended purpose.

Machi could use Riak Ensemble or ZooKeeper as an oracle (or perhaps as a oracle
coordinator), but we wish to keep Machi free of big external
dependencies.  We would also like to see Machi be able to
operate in an ``AP mode'', which means providing service even
if all network communication to an oracle is broken.

The model of projection calculation and storage described in
Section~\ref{sec:projections} allows for each server to operate
independently, if necessary.  This autonomy allows the server in AP
mode to
always accept new writes: new writes are written to unique file names
and unique file offsets using a chain consisting of only a single FLU,
if necessary.  How is this possible?  Let's look at a scenario in
Section~\ref{sub:split-brain-scenario}.

\subsection{A split brain scenario}
\label{sub:split-brain-scenario}

\begin{enumerate}

\item Assume 3 Machi FLUs, all in good health and perfect data sync: $[F_a,
  F_b, F_c]$ using projection epoch $P_p$.

\item Assume data $D_0$ is written at offset $O_0$ in Machi file
  $F_0$.

\item Then a network partition happens.  Servers $F_a$ and $F_b$ are
  on one side of the split, and server $F_c$ is on the other side of
  the split.  We'll call them the ``left side'' and ``right side'',
  respectively.

\item On the left side, $F_b$ calculates a new projection and writes
  it unanimously (to two projection stores) as epoch $P_B+1$.  The
  subscript $_B$ denotes a
  version of projection epoch $P_{p+1}$ that was created by server $F_B$
  and has a unique checksum (used to detect differences after the
  network partition heals).

\item In parallel, on the right side, $F_c$ calculates a new
  projection and writes it unanimously (to a single projection store)
  as epoch $P_c+1$.

\item In parallel, a client on the left side writes data $D_1$
  at offset $O_1$ in Machi file $F_1$, and also
  a client on the right side writes data $D_2$
  at offset $O_2$ in Machi file $F_2$.  We know that $F_1 \ne F_2$
  because each sequencer is forced to choose disjoint filenames from
  any prior epoch whenever a new projection is available.

\end{enumerate}

Now, what happens when various clients attempt to read data values
$D_0$, $D_1$, and $D_2$?

\begin{itemize}
\item All clients can read $D_0$.
\item Clients on the left side can read $D_1$.
\item Attempts by clients on the right side to read $D_1$ will get
  {\tt error\_unavailable}.
\item Clients on the right side can read $D_2$.
\item Attempts by clients on the left side to read $D_2$ will get
  {\tt error\_unavailable}.
\end{itemize}

The {\tt error\_unavailable} result is not an error in the CAP Theorem
sense: it is a valid and affirmative response.  In both cases, the
system on the client's side definitely knows that the cluster is
partitioned.  If Machi were not a write-once store, perhaps there
might be an old/stale value to read on the local side of the network
partition \ldots but the system also knows definitely that no
old/stale value exists.  Therefore Machi remains available in the
CAP Theorem sense both for writes and reads.

We know that all files $F_0$,
$F_1$, and $F_2$ are disjoint and can be merged (in a manner analogous
to set union) onto each server in $[F_a, F_b, F_c]$ safely
when the network partition is healed.  However,
unlike pure theoretical set union, Machi's data merge \& repair
operations must operate within some constraints that are designed to
prevent data loss.

\subsection{Aside: defining data availability and data loss}
\label{sub:define-availability}

Let's take a moment to be clear about definitions:

\begin{itemize}
\item ``data is available at time $T$'' means that data is available
  for reading at $T$: the Machi cluster knows for certain that the
  requested data is not been written or it is written and has a single
  value.
\item ``data is unavailable at time $T$'' means that data is
  unavailable for reading at $T$ due to temporary circumstances,
  e.g. network partition.  If a read request is issued at some time
  after $T$, the data will be available.
\item ``data is lost at time $T$'' means that data is permanently
  unavailable at $T$ and also all times after $T$.
\end{itemize}

Chain Replication is a fantastic technique for managing the
consistency of data across a number of whole replicas.  There are,
however, cases where CR can indeed lose data.

\subsection{Data loss scenario \#1: too few servers}
\label{sub:data-loss1}

If the chain is $N$ servers long, and if all $N$ servers fail, then
of course data is unavailable.  However, if all $N$ fail
permanently, then data is lost.

If the administrator had intended to avoid data loss after $N$
failures, then the administrator would have provisioned a Machi
cluster with at least $N+1$ servers.

\subsection{Data Loss scenario \#2: bogus configuration change sequence}
\label{sub:data-loss2}

Assume that the sequence of events in Figure~\ref{fig:data-loss2} takes place.

\begin{figure}
\begin{enumerate}
%% NOTE: the following list 9 items long.  We use that fact later, see
%% string YYY9 in a comment further below.  If the length of this list
%% changes, then the counter reset below needs adjustment.
\item Projection $P_p$ says that chain membership is $[F_a]$.
\item A write of data $D$ to file $F$ at offset $O$ is successful.
\item Projection $P_{p+1}$ says that chain membership is $[F_a,F_b]$, via
   an administration API request.
\item Machi will trigger repair operations, copying any missing data
   files from FLU $F_a$ to FLU $F_b$.  For the purpose of this
   example, the sync operation for file $F$'s data and metadata has
   not yet started.
\item FLU $F_a$ crashes.
\item The chain manager on $F_b$ notices $F_a$'s crash,
   decides to create a new projection $P_{p+2}$ where chain membership is
   $[F_b]$
  successfully stores $P_{p+2}$ in its local store.  FLU $F_b$ is now wedged.
\item FLU $F_a$ is down, therefore the
   value of $P_{p+2}$ is unanimous for all currently available FLUs
   (namely $[F_b]$).
\item FLU $F_b$ sees that projection $P_{p+2}$ is the newest unanimous
   projection.  It unwedges itself and continues operation using $P_{p+2}$.
\item Data $D$ is definitely unavailable for now, perhaps lost forever?
\end{enumerate}
\caption{Data unavailability scenario with danger of permanent data loss}
\label{fig:data-loss2}
\end{figure}

At this point, the data $D$ is not available on $F_b$.  However, if
we assume that $F_a$ eventually returns to service, and Machi
correctly acts to repair all data within its chain, then $D$
all of its contents will be available eventually.

However, if server $F_a$ never returns to service, then $D$ is lost.  The
Machi administration API must always warn the user that data loss is
possible.  In Figure~\ref{fig:data-loss2}'s scenario, the API must
warn the administrator in multiple ways that fewer than the full {\tt
  length(all\_members)} number of replicas are in full sync.

A careful reader should note that $D$ is also lost if step \#5 were
instead, ``The hardware that runs FLU $F_a$ was destroyed by fire.''
For any possible step following \#5, $D$ is lost.  This is data loss
for the same reason that the scenario of Section~\ref{sub:data-loss1}
happens: the administrator has not provisioned a sufficient number of
replicas.

Let's revisit Figure~\ref{fig:data-loss2}'s scenario yet again.  This
time, we add a final step at the end of the sequence:

\begin{enumerate}
\setcounter{enumi}{9}           % YYY9
\item The administration API is used to change the chain
configuration to {\tt all\_members=$[F_b]$}.
\end{enumerate}

Step \#10 causes data loss.  Specifically, the only copy of file
$F$ is on FLU $F_a$.  By administration policy, FLU $F_a$ is now
permanently inaccessible.

The chain manager {\em must} keep track of all
repair operations and their status.  If such information is tracked by
all FLUs, then the data loss by bogus administrator action can be
prevented.  In this scenario, FLU $F_b$ knows that `$F_a \rightarrow
F_b$` repair has not yet finished and therefore it is unsafe to remove
$F_a$ from the cluster.

\subsection{Data Loss scenario \#3: chain replication repair done badly}
\label{sub:data-loss3}

It's quite possible to lose data through careless/buggy Chain
Replication chain configuration changes.  For example, in the split
brain scenario of Section~\ref{sub:split-brain-scenario}, we have two
pieces of data written to different ``sides'' of the split brain,
$D_0$ and $D_1$.  If the chain is naively reconfigured after the network
partition heals to be $[F_a=\emptyset,F_b=\emptyset,F_c=D_1],$ then $D_1$
is in danger of being lost.  Why?
The Update Propagation Invariant is violated.
Any Chain Replication read will be
directed to the tail, $F_c$.  The value exists there, so there is no
need to do any further work; the unwritten values at $F_a$ and $F_b$
will not be repaired.  If the $F_c$ server fails sometime
later, then $D_1$ will be lost.  The ``Chain Replication Repair''
section of \cite{machi-design} discusses
how data loss can be avoided after servers are added (or re-added) to
an active chain configuration.

\subsection{Summary}

We believe that maintaining the Update Propagation Invariant is a
hassle anda pain, but that hassle and pain are well worth the
sacrifices required to maintain the invariant at all times.  It avoids
data loss in all cases where the U.P.~Invariant preserving chain
contains at least one FLU.

\section{Chain Replication: proof of correctness}
\label{sec:cr-proof}

See Section~3 of \cite{chain-replication} for a proof of the
correctness of Chain Replication.  A short summary is provide here.
Readers interested in good karma should read the entire paper.

\subsection{The Update Propagation Invariant}
\label{sub:upi}

``Update Propagation Invariant'' is the original chain replication
paper's name for the
$H_i \succeq H_j$
property mentioned in Figure~\ref{tab:chain-order}.
This paper will use the same name.
This property may also be referred to by its acronym, ``UPI''.

\subsection{Chain Replication and strong consistency}

The three basic rules of Chain Replication and its strong
consistency guarantee:

\begin{enumerate}

\item All replica servers are arranged in an ordered list $C$.

\item All mutations of a datum are performed upon each replica of $C$
  strictly in the order which they appear in $C$.  A mutation is considered
  completely successful if the writes by all replicas are successful.

\item The head of the chain makes the determination of the order of
  all mutations to all members of the chain.  If the head determines
  that some mutation $M_i$ happened before another mutation $M_j$,
  then mutation $M_i$ happens before $M_j$ on all other members of
  the chain.\footnote{While necesary for general Chain Replication,
    Machi does not need this property.  Instead, the property is
    provided by Machi's sequencer and the write-once register of each
    byte in each file.}

\item All read-only operations are performed by the ``tail'' replica,
  i.e., the last replica in $C$.

\end{enumerate}

The basis of the proof lies in a simple logical trick, which is to
consider the history of all operations made to any server in the chain
as a literal list of unique symbols, one for each mutation.

Each replica of a datum will have a mutation history list.  We will
call this history list $H$. For the $i^{th}$ replica in the chain list
$C$, we call $H_i$ the mutation history list for the $i^{th}$ replica.

Before the $i^{th}$ replica in the chain list begins service, its mutation
history $H_i$ is empty, $[]$.  After this replica runs in a Chain
Replication system for a while, its mutation history list grows to
look something like 
$[M_0, M_1, M_2, ..., M_{m-1}]$ where $m$ is the total number of
mutations of the datum that this server has processed successfully.

Let's assume for a moment that all mutation operations have stopped.
If the order of the chain was constant, and if all mutations are
applied to each replica in the chain's order, then all replicas of a
datum will have the exact same mutation history: $H_i = H_J$ for any
two replicas $i$ and $j$ in the chain
(i.e., $\forall i,j \in C, H_i = H_J$).  That's a lovely property,
but it is much more interesting to assume that the service is
not stopped.  Let's look next at a running system.

\begin{figure*}
\centering
\begin{tabular}{ccc}
{\bf {{On left side of $C$}}} & & {\bf On right side of $C$} \\
\hline
\multicolumn{3}{l}{Looking at replica order in chain $C$:} \\
$i$ & $<$ & $j$ \\

\multicolumn{3}{l}{For example:} \\

0 & $<$ & 2 \\
\hline
\multicolumn{3}{l}{It {\em must} be true: history lengths per replica:} \\
length($H_i$) & $\geq$ & length($H_j$) \\
\multicolumn{3}{l}{For example, a quiescent chain:} \\
length($H_i$) = 48 & $\geq$ & length($H_j$) = 48 \\
\multicolumn{3}{l}{For example, a chain being mutated:} \\
length($H_i$) = 55 & $\geq$ & length($H_j$) = 48 \\
\multicolumn{3}{l}{Example ordered mutation sets:} \\
$[M_0,M_1,\ldots,M_{46},M_{47},\ldots,M_{53},M_{54}]$ & $\supset$ & $[M_0,M_1,\ldots,M_{46},M_{47}]$ \\
\multicolumn{3}{c}{\bf Therefore the right side is always an ordered
  subset} \\
\multicolumn{3}{c}{\bf of the left side.  Furthermore, the ordered
  sets on both} \\
\multicolumn{3}{c}{\bf sides have the exact same order of those elements they have in common.} \\
\multicolumn{3}{c}{The notation used by the Chain Replication paper is
shown below:} \\
$[M_0,M_1,\ldots,M_{46},M_{47},\ldots,M_{53},M_{54}]$ & $\succeq$ & $[M_0,M_1,\ldots,M_{46},M_{47}]$ \\

\end{tabular}
\caption{A demonstration of Chain Replication protocol history ``Update Propagation Invariant''.}
\label{tab:chain-order}
\end{figure*}

If the entire chain $C$ is processing any number of concurrent
mutations, then we can still understand $C$'s behavior.
Figure~\ref{tab:chain-order} shows us two replicas in chain $C$:
replica $R_i$ that's on the left/earlier side of the replica chain $C$
than some other replica $R_j$.  We know that $i$'s position index in
the chain is smaller than $j$'s position index, so therefore $i < j$.
The restrictions of Chain Replication make it true that length($H_i$)
$\ge$ length($H_j$) because it's also that $H_i \supset H_j$, i.e,
$H_i$ on the left is always is a superset of $H_j$ on the right.

When considering $H_i$ and $H_j$ as strictly ordered lists, we have 
$H_i \succeq H_j$, where the right side is always an exact prefix of the left
side's list.  This prefixing propery is exactly what strong
consistency requires.  If a value is read from the tail of the chain,
then no other chain member can have a prior/older value because their
respective mutations histories cannot be shorter than the tail
member's history.

\section{Repair of entire files}
\label{sec:repair-entire-files}

There are some situations where repair of entire files is necessary.

\begin{itemize}
\item To repair servers added to a chain in a projection change,
  specifically adding a new server to the chain.  This case covers both
  adding a new, data-less server and re-adding a previous, data-full server
  back to the chain.
\item To avoid data loss when changing the order of the chain's servers.
\end{itemize}

Both situations can set the stage for data loss in the future.
If a violation of the Update Propagation Invariant (see end of
Section~\ref{sec:cr-proof}) is permitted, then the strong consistency
guarantee of Chain Replication is violated.  Because Machi uses
write-once registers, the number of possible strong consistency
violations is small: any client that witnesses a written $\rightarrow$
unwritten transition is a violation of strong consistency.  But
avoiding even this one bad scenario is a bit tricky.

As explained in Section~\ref{sub:data-loss1}, data
unavailability/loss when all chain servers fail is unavoidable.  We
wish to avoid data loss whenever a chain has at least one surviving
server.  Another method to avoid data loss is to preserve the Update
Propagation Invariant at all times.

\subsection{Just ``rsync'' it!}
\label{ssec:just-rsync-it}

A simple repair method might be perhaps 90\% sufficient.
That method could loosely be described as ``just {\tt rsync}
all files to all servers in an infinite loop.''\footnote{The
  file format suggested in
  \cite{machi-design} does not permit {\tt rsync}
  as-is to be sufficient.  A variation of {\tt rsync} would need to be
  aware of the data/metadata split within each file and only replicate
  the data section \ldots and the metadata would still need to be
  managed outside of {\tt rsync}.}

However, such an informal method
cannot tell you exactly when you are in danger of data loss and when
data loss has actually happened.  If we maintain the Update
Propagation Invariant, then we know exactly when data loss is immanent
or has happened.

Furthermore, we hope to use Machi for multiple use cases, including
ones that require strong consistency.
For uses such as CORFU, strong consistency is a non-negotiable
requirement.  Therefore, we will use the Update Propagation Invariant
as the foundation for Machi's data loss prevention techniques.

\subsection{Divergence from CORFU: repair}
\label{sub:repair-divergence}

The original repair design for CORFU is simple and effective,
mostly.  See Figure~\ref{fig:corfu-style-repair} for a full
description of the algorithm
Figure~\ref{fig:corfu-repair-sc-violation} for an example of a strong
consistency violation that can follow.  (NOTE: This is a variation of
the data loss scenario that is described in
Figure~\ref{fig:data-loss2}.)

\begin{figure}
\begin{enumerate}
\item Destroy all data on the repair destination FLU.
\item Add the repair destination FLU to the tail of the chain in a new
  projection $P_{p+1}$.
\item Change projection from $P_p$ to $P_{p+1}$.
\item Let single item read repair fix all of the problems.
\end{enumerate}
\caption{Simplest CORFU-style repair algorithm.}
\label{fig:corfu-style-repair}
\end{figure}

\begin{figure}
\begin{enumerate}
\item Write value $V$ to offset $O$ in the log with chain $[F_a]$.
  This write is considered successful.
\item Change projection to configure chain as $[F_a,F_b]$.  Prior to
  the change, all values on FLU $F_b$ are unwritten.
\item FLU server $F_a$ crashes.  The new projection defines the chain
  as $[F_b]$.
\item A client attempts to read offset $O$ and finds an unwritten
  value.  This is a strong consistency violation.
%% \item The same client decides to fill $O$ with the junk value
%%   $V_{junk}$.  Now value $V$ is lost.
\end{enumerate}
\caption{An example scenario where the CORFU simplest repair algorithm
  can lead to a violation of strong consistency.}
\label{fig:corfu-repair-sc-violation}
\end{figure}

A variation of the repair
algorithm is presented in section~2.5 of a later CORFU paper \cite{corfu2}.
However, the re-use a failed
server is not discussed there, either: the example of a failed server
$F_6$ uses a new server, $F_8$ to replace $F_6$.  Furthermore, the
repair process is described as:

\begin{quote}
``Once $F_6$ is completely rebuilt on $F_8$ (by copying entries from
  $F_7$), the system moves to projection (C), where $F_8$ is now used
  to service all reads in the range $[40K,80K)$.''
\end{quote}

The phrase ``by copying entries'' does not give enough
detail to avoid the same data race as described in
Figure~\ref{fig:corfu-repair-sc-violation}.  We believe that if
``copying entries'' means copying only written pages, then CORFU
remains vulnerable.  If ``copying entries'' also means ``fill any
unwritten pages prior to copying them'', then perhaps the
vulnerability is eliminated.\footnote{SLF's note: Probably?  This is my
  gut feeling right now.  However, given that I've just convinced
  myself 100\% that fill during any possibility of split brain is {\em
  not safe} in Machi, I'm not 100\% certain anymore than this ``easy''
  fix for CORFU is correct.}.

\subsection{Whole-file repair as FLUs are (re-)added to a chain}
\label{sub:repair-add-to-chain}

Machi's repair process must preserve the Update Propagation
Invariant.  To avoid data races with data copying from
``U.P.~Invariant preserving'' servers (i.e. fully repaired with
respect to the Update Propagation Invariant)
to servers of unreliable/unknown state, a
projection like the one shown in
Figure~\ref{fig:repair-chain-of-chains} is used.  In addition, the
operations rules for data writes and reads must be observed in a
projection of this type.

\begin{figure*}
\centering
$
[\overbrace{\underbrace{H_1}_\textbf{Head of Heads}, M_{11},
      \underbrace{T_1}_\textbf{Tail \#1}}^\textbf{Chain \#1 (U.P.~Invariant preserving)}
\mid
\overbrace{H_2, M_{21},
      \underbrace{T_2}_\textbf{Tail \#2}}^\textbf{Chain \#2 (repairing)}
\mid \ldots \mid
\overbrace{H_n, M_{n1},
      \underbrace{T_n}_\textbf{Tail \#n \& Tail of Tails ($T_{tails}$)}}^\textbf{Chain \#n (repairing)}
]
$
\caption{Representation of a ``chain of chains'': a chain prefix of
  Update Propagation Invariant preserving FLUs (``Chain \#1'')
  with FLUs from $n-1$ other chains under repair.}
\label{fig:repair-chain-of-chains}
\end{figure*}

\begin{itemize}

\item The system maintains the distinction between ``U.P.~preserving''
  and ``repairing'' FLUs at all times.  This allows the system to
  track exactly which servers are known to preserve the Update
  Propagation Invariant and which servers may/may not.

\item All ``repairing'' FLUs must be added only at the end of the
  chain-of-chains.

\item All write operations must flow successfully through the
  chain-of-chains from beginning to end, i.e., from the ``head of
  heads'' to the ``tail of tails''.  This rule also includes any
  repair operations.

\item In AP Mode, all read operations are attempted from the list of
$[T_1,\-T_2,\-\ldots,\-T_n]$, where these FLUs are the tails of each of the
chains involved in repair.
In CP mode, all read operations are attempted only from $T_1$.
The first reply of {\tt \{ok, <<...>>\}} is a correct answer;
the rest of the FLU list can be ignored and the result returned to the
client.  If all FLUs in the list have an unwritten value, then the
client can return {\tt error\_unwritten}.

\end{itemize}

While the normal single-write and single-read operations are performed
by the cluster, a file synchronization process is initiated.  The
sequence of steps differs depending on the AP or CP mode of the system.

\subsubsection{Cluster in CP mode}

In cases where the cluster is operating in CP Mode,
CORFU's repair method of ``just copy it all'' (from source FLU to repairing
FLU) is correct, {\em except} for the small problem pointed out in
Section~\ref{sub:repair-divergence}.  The problem for Machi is one of
time \& space.  Machi wishes to avoid transferring data that is
already correct on the repairing nodes.  If a Machi node is storing
20TBytes of data, we really do not wish to use 20TBytes of bandwidth
to repair only 1 GByte of truly-out-of-sync data.

However, it is {\em vitally important} that all repairing FLU data be
clobbered/overwritten with exactly the same data as the Update
Propagation Invariant preserving chain.  If this rule is not strictly
enforced, then fill operations can corrupt Machi file data.  The
algorithm proposed is:

\begin{enumerate}

\item Change the projection to a ``chain of chains'' configuration
  such as depicted in Figure~\ref{fig:repair-chain-of-chains}.

\item For all files on all FLUs in all chains, extract the lists of
  written/unwritten byte ranges and their corresponding file data
  checksums.  (The checksum metadata is not strictly required for
  recovery in AP Mode.)
  Send these lists to the tail of tails
  $T_{tails}$, which will collate all of the lists into a list of
  tuples such as {\tt \{FName, $O_{start}, O_{end}$, CSum, FLU\_List\}}
  where {\tt FLU\_List} is the list of all FLUs in the entire chain of
  chains where the bytes at the location {\tt \{FName, $O_{start},
    O_{end}$\}} are known to be written (as of the current repair period).

\item For chain \#1 members, i.e., the
  leftmost chain relative to Figure~\ref{fig:repair-chain-of-chains},
  repair files byte ranges for any chain \#1 members that are not members
  of the {\tt FLU\_List} set.  This will repair any partial
  writes to chain \#1 that were unsuccessful (e.g., client crashed).
  (Note however that this step only repairs FLUs in chain \#1.)

\item For all file byte ranges in all files on all FLUs in all
  repairing chains where Tail \#1's value is unwritten, force all
  repairing FLUs to also be unwritten.

\item For file byte ranges in all files on all FLUs in all repairing
  chains where Tail \#1's value is written, send repair file byte data
  \& metadata to any repairing FLU if the value repairing FLU's
  value is unwritten or the checksum is not exactly equal to Tail \#1's
  checksum.

\end{enumerate}

\begin{figure}
\centering
$
[\overbrace{\underbrace{H_1}_\textbf{Head}, M_{11}, T_1,
                        H_2, M_{21}, T_2,
                        \ldots
                        H_n, M_{n1},
                        \underbrace{T_n}_\textbf{Tail}}^\textbf{Chain (U.P.~Invariant preserving)}
]
$
\caption{Representation of Figure~\ref{fig:repair-chain-of-chains}
  after all repairs have finished successfully and a new projection has
  been calculated.}
\label{fig:repair-chain-of-chains-finished}
\end{figure}

When the repair is known to have copied all missing data successfully,
then the chain can change state via a new projection that includes the
repaired FLU(s) at the end of the U.P.~Invariant preserving chain \#1
in the same order in which they appeared in the chain-of-chains during
repair.  See Figure~\ref{fig:repair-chain-of-chains-finished}.

The repair can be coordinated and/or performed by the $T_{tails}$ FLU
or any other FLU or cluster member that has spare capacity.

There is no serious race condition here between the enumeration steps
and the repair steps.  Why?  Because the change in projection at
step \#1 will force any new data writes to adapt to a new projection.
Consider the mutations that either happen before or after a projection
change:


\begin{itemize}

\item For all mutations $M_1$ prior to the projection change, the
  enumeration steps \#3 \& \#4 and \#5 will always encounter mutation
  $M_1$.  Any repair must write through the entire chain-of-chains and
  thus will preserve the Update Propagation Invariant when repair is
  finished.

\item For all mutations $M_2$ starting during or after the projection
  change has finished, a new mutation $M_2$ may or may not be included in the
  enumeration steps \#3 \& \#4 and \#5.
  However, in the new projection, $M_2$ must be
  written to all chain of chains members, and such
  in-order writes will also preserve the Update
  Propagation Invariant and therefore is also be safe.

\end{itemize}

\subsubsection{Cluster in AP Mode}

In cases the cluster is operating in AP Mode:

\begin{enumerate}
\item Follow the first two steps of the ``CP Mode''
  sequence (above).
\item Follow step \#3 of the ``strongly consistent mode'' sequence
  (above), but in place of repairing only FLUs in Chain \#1, AP mode
  will repair the byte range of any FLU that is not a member of the
  {\tt FLU\_List} set.
\item End of procedure.
\end{enumerate}

The end result is a huge ``merge'' where any
{\tt \{FName, $O_{start}, O_{end}$\}} range of bytes that is written
on FLU $F_w$ but missing/unwritten from FLU $F_m$ is written down the full chain
of chains, skipping any FLUs where the data is known to be written.
Such writes will also preserve Update Propagation Invariant when
repair is finished.

\subsection{Whole-file repair when changing FLU ordering within a chain}
\label{sub:repair-chain-re-ordering}

Changing FLU order within a chain is an operations optimization only.
It may be that the administrator wishes the order of a chain to remain
as originally configured during steady-state operation, e.g.,
$[F_a,F_b,F_c]$.  As FLUs are stopped \& restarted, the chain may
become re-ordered in a seemingly-arbitrary manner.

It is certainly possible to re-order the chain, in a kludgy manner.
For example, if the desired order is $[F_a,F_b,F_c]$ but the current
operating order is $[F_c,F_b,F_a]$, then remove $F_b$ from the chain,
then add $F_b$ to the end of the chain.  Then repeat the same
procedure for $F_c$.  The end result will be the desired order.

From an operations perspective, re-ordering of the chain
using this kludgy manner has a
negative effect on availability: the chain is temporarily reduced from
operating with $N$ replicas down to $N-1$.  This reduced replication
factor will not remain for long, at most a few minutes at a time, but
even a small amount of time may be unacceptable in some environments.

Reordering is possible with the introduction of a ``temporary head''
of the chain.  This temporary FLU does not need to be a full replica
of the entire chain --- it merely needs to store replicas of mutations
that are made during the chain reordering process.  This method will
not be described here.  However, {\em if reviewers believe that it should
be included}, please let the authors know.

\subsubsection{In both Machi operating modes:}
After initial implementation, it may be that the repair procedure is a
bit too slow.  In order to accelerate repair decisions, it would be
helpful have a quicker method to calculate which files have exactly
the same contents.  In traditional systems, this is done with a single
file checksum; see also the ``checksum scrub'' subsection in
\cite{machi-design}.
Machi's files can be written out-of-order from a file offset point of
view, which violates the order which the traditional method for
calculating a full-file hash.  If we recall out-of-temporal-order
example in the ``Append-only files'' section of \cite{machi-design},
the traditional method cannot
continue calculating the file checksum at offset 2 until the byte at
file offset 1 is written.

It may be advantageous for each FLU to maintain for each file a
checksum of a canonical representation of the
{\tt \{$O_{start},O_{end},$ CSum\}} tuples that the FLU must already
maintain.  Then for any two FLUs that claim to store a file $F$, if
both FLUs have the same hash of $F$'s written map + checksums, then
the copies of $F$ on both FLUs are the same.

\bibliographystyle{abbrvnat}
\begin{thebibliography}{}
\softraggedright

\bibitem{riak-core}
Klophaus, Rusty.
"Riak Core."
ACM SIGPLAN Commercial Users of Functional Programming (CUFP'10), 2010.
{\tt http://dl.acm.org/citation.cfm?id=1900176} and
{\tt https://github.com/basho/riak\_core}

\bibitem{rfc-7282}
RFC 7282: On Consensus and Humming in the IETF.
Internet Engineering Task Force.
{\tt https://tools.ietf.org/html/rfc7282}

\bibitem{elastic-chain-replication}
Abu-Libdeh, Hussam et al.
Leveraging Sharding in the Design of Scalable Replication Protocols.
Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC'13), 2013.
{\tt http://www.ymsir.com/papers/sharding-socc.pdf}

\bibitem{corfu1}
Balakrishnan, Mahesh et al.
CORFU: A Shared Log Design for Flash Clusters.
Proceedings of the 9th USENIX Conference on Networked Systems Design
and Implementation (NSDI'12), 2012.
{\tt http://research.microsoft.com/pubs/157204/ corfumain-final.pdf}

\bibitem{corfu2}
Balakrishnan, Mahesh et al.
CORFU: A Distributed Shared Log
ACM Transactions on Computer Systems, Vol. 31, No. 4, Article 10, December 2013.
{\tt http://www.snookles.com/scottmp/corfu/ corfu.a10-balakrishnan.pdf}

\bibitem{machi-design}
Basho Japan KK.
Machi: an immutable file store
{\tt https://github.com/basho/machi/tree/ master/doc/high-level-machi.pdf}

\bibitem{was}
Calder, Brad et al.
Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency
Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP'11), 2011.
{\tt http://sigops.org/sosp/sosp11/current/ 2011-Cascais/printable/11-calder.pdf}

\bibitem{cr-theory-and-practice}
Fritchie, Scott Lystig.
Chain Replication in Theory and in Practice.
Proceedings of the 9th ACM SIGPLAN Workshop on Erlang (Erlang'10), 2010.
{\tt http://www.snookles.com/scott/publications/ erlang2010-slf.pdf}

\bibitem{humming-consensus-allegory}
Fritchie, Scott Lystig.
On “Humming Consensus”, an allegory.
{\tt http://www.snookles.com/slf-blog/2015/03/ 01/on-humming-consensus-an-allegory/}

\bibitem{the-log-what}
Kreps, Jay.
The Log: What every software engineer should know about real-time data's unifying abstraction
{\tt http://engineering.linkedin.com/distributed-
  systems/log-what-every-software-engineer-should-
  know-about-real-time-datas-unifying}

\bibitem{kafka}
Kreps, Jay et al.
Kafka: a distributed messaging system for log processing.
NetDB’11.
{\tt http://research.microsoft.com/en-us/UM/people/
  srikanth/netdb11/netdb11papers/netdb11-final12.pdf}

\bibitem{part-time-parliament}
Lamport, Leslie.
The Part-Time Parliament.
DEC technical report SRC-049, 1989.
{\tt ftp://apotheca.hpl.hp.com/gatekeeper/pub/ DEC/SRC/research-reports/SRC-049.pdf}

\bibitem{paxos-made-simple}
Lamport, Leslie.
Paxos Made Simple.
In SIGACT News \#4, Dec, 2001.
{\tt http://research.microsoft.com/users/ lamport/pubs/paxos-simple.pdf}

\bibitem{random-slicing}
Miranda, Alberto et al.
Random Slicing: Efficient and Scalable Data Placement for Large-Scale Storage Systems.
ACM Transactions on Storage, Vol. 10, No. 3, Article 9, July 2014.
{\tt http://www.snookles.com/scottmp/corfu/random- slicing.a9-miranda.pdf}

\bibitem{porcupine}
Saito, Yasushi et al.
Manageability, availability and performance in Porcupine: a highly scalable, cluster-based mail service.
7th ACM Symposium on Operating System Principles (SOSP’99).
{\tt http://homes.cs.washington.edu/\%7Elevy/ porcupine.pdf}

\bibitem{chain-replication}
van Renesse, Robbert et al.
Chain Replication for Supporting High Throughput and Availability.
Proceedings of the 6th Conference on Symposium on Operating Systems
Design \& Implementation (OSDI'04) - Volume 6, 2004.
{\tt http://www.cs.cornell.edu/home/rvr/papers/ osdi04.pdf}

\end{thebibliography}


\end{document}