62d3dadf98
All of the major surgery required to move Chain Manager design & discussion details out of the high-level-machi.tex document are complete. I've done only a very small amount of work on the original high-level-machi.tex to fix document flow problems. There's probably a good way to have LaTeX automatically manage the mutual references between the now-split documents, but I didn't know about, sorry.
1186 lines
50 KiB
TeX
1186 lines
50 KiB
TeX
|
||
%% \documentclass[]{report}
|
||
\documentclass[preprint,10pt]{sigplanconf}
|
||
% The following \documentclass options may be useful:
|
||
|
||
% preprint Remove this option only once the paper is in final form.
|
||
% 10pt To set in 10-point type instead of 9-point.
|
||
% 11pt To set in 11-point type instead of 9-point.
|
||
% authoryear To obtain author/year citation style instead of numeric.
|
||
|
||
% \usepackage[a4paper]{geometry}
|
||
\usepackage[dvips]{graphicx} % to include images
|
||
%\usepackage{pslatex} % to use PostScript fonts
|
||
|
||
\begin{document}
|
||
|
||
%%\special{papersize=8.5in,11in}
|
||
%%\setlength{\pdfpageheight}{\paperheight}
|
||
%%\setlength{\pdfpagewidth}{\paperwidth}
|
||
|
||
\conferenceinfo{}{}
|
||
\copyrightyear{2014}
|
||
\copyrightdata{978-1-nnnn-nnnn-n/yy/mm}
|
||
\doi{nnnnnnn.nnnnnnn}
|
||
|
||
\titlebanner{Draft \#0, April 2014}
|
||
\preprintfooter{Draft \#0, April 2014}
|
||
|
||
\title{Machi Chain Replication: management theory and design}
|
||
\subtitle{}
|
||
|
||
\authorinfo{Basho Japan KK}{}
|
||
|
||
\maketitle
|
||
|
||
\section{Origins}
|
||
\label{sec:origins}
|
||
|
||
This document was first written during the autumn of 2014 for a
|
||
Basho-only internal audience. Since its original drafts, Machi has
|
||
been designated by Basho as a full open source software project. This
|
||
document has been rewritten in 2015 to address an external audience.
|
||
For an overview of the design of the larger Machi system, please see
|
||
\cite{machi-design}.
|
||
|
||
\section{Abstract}
|
||
\label{sec:abstract}
|
||
|
||
TODO
|
||
|
||
\section{Introduction}
|
||
\label{sec:introduction}
|
||
|
||
TODO
|
||
|
||
\section{Projections: calculation, then storage, then (perhaps) use}
|
||
\label{sec:projections}
|
||
|
||
Machi uses a ``projection'' to determine how its Chain Replication replicas
|
||
should operate; see \cite{machi-design} and
|
||
\cite{corfu1}. At runtime, a cluster must be able to respond both to
|
||
administrative changes (e.g., substituting a failed server box with
|
||
replacement hardware) as well as local network conditions (e.g., is
|
||
there a network partition?). The concept of a projection is borrowed
|
||
from CORFU but has a longer history, e.g., the Hibari key-value store
|
||
\cite{cr-theory-and-practice} and goes back in research for decades,
|
||
e.g., Porcupine \cite{porcupine}.
|
||
|
||
\subsection{Phases of projection change}
|
||
|
||
Machi's use of projections is in four discrete phases and are
|
||
discussed below: network monitoring,
|
||
projection calculation, projection storage, and
|
||
adoption of new projections.
|
||
|
||
\subsubsection{Network monitoring}
|
||
\label{sub:network-monitoring}
|
||
|
||
Monitoring of local network conditions can be implemented in many
|
||
ways. None are mandatory, as far as this RFC is concerned.
|
||
Easy-to-maintain code should be the primary driver for any
|
||
implementation. Early versions of Machi may use some/all of the
|
||
following techniques:
|
||
|
||
\begin{itemize}
|
||
\item Internal ``no op'' FLU-level protocol request \& response.
|
||
\item Use of distributed Erlang {\tt net\_ticktime} node monitoring
|
||
\item Explicit connections of remote {\tt epmd} services, e.g., to
|
||
tell the difference between a dead Erlang VM and a dead
|
||
machine/hardware node.
|
||
\item Network tests via ICMP {\tt ECHO\_REQUEST}, a.k.a. {\tt ping(8)}
|
||
\end{itemize}
|
||
|
||
Output of the monitor should declare the up/down (or
|
||
available/unavailable) status of each server in the projection. Such
|
||
Boolean status does not eliminate ``fuzzy logic'' or probabilistic
|
||
methods for determining status. Instead, hard Boolean up/down status
|
||
decisions are required by the projection calculation phase
|
||
(Section~\ref{subsub:projection-calculation}).
|
||
|
||
\subsubsection{Projection data structure calculation}
|
||
\label{subsub:projection-calculation}
|
||
|
||
Each Machi server will have an independent agent/process that is
|
||
responsible for calculating new projections. A new projection may be
|
||
required whenever an administrative change is requested or in response
|
||
to network conditions (e.g., network partitions).
|
||
|
||
Projection calculation will be a pure computation, based on input of:
|
||
|
||
\begin{enumerate}
|
||
\item The current projection epoch's data structure
|
||
\item Administrative request (if any)
|
||
\item Status of each server, as determined by network monitoring
|
||
(Section~\ref{sub:network-monitoring}).
|
||
\end{enumerate}
|
||
|
||
All decisions about {\em when} to calculate a projection must be made
|
||
using additional runtime information. Administrative change requests
|
||
probably should happen immediately. Change based on network status
|
||
changes may require retry logic and delay/sleep time intervals.
|
||
|
||
\subsection{Projection storage: writing}
|
||
\label{sub:proj-storage-writing}
|
||
|
||
All projection data structures are stored in the write-once Projection
|
||
Store that is run by each FLU. (See also \cite{machi-design}.)
|
||
|
||
Writing the projection follows the two-step sequence below.
|
||
In cases of writing
|
||
failure at any stage, the process is aborted. The most common case is
|
||
{\tt error\_written}, which signifies that another actor in the system has
|
||
already calculated another (perhaps different) projection using the
|
||
same projection epoch number and that
|
||
read repair is necessary. Note that {\tt error\_written} may also
|
||
indicate that another actor has performed read repair on the exact
|
||
projection value that the local actor is trying to write!
|
||
|
||
\begin{enumerate}
|
||
\item Write $P_{new}$ to the local projection store. This will trigger
|
||
``wedge'' status in the local FLU, which will then cascade to other
|
||
projection-related behavior within the FLU.
|
||
\item Write $P_{new}$ to the remote projection store of {\tt all\_members}.
|
||
Some members may be unavailable, but that is OK.
|
||
\end{enumerate}
|
||
|
||
(Recall: Other parts of the system are responsible for reading new
|
||
projections from other actors in the system and for deciding to try to
|
||
create a new projection locally.)
|
||
|
||
\subsection{Projection storage: reading}
|
||
\label{sub:proj-storage-reading}
|
||
|
||
Reading data from the projection store is similar in principle to
|
||
reading from a Chain Replication-managed FLU system. However, the
|
||
projection store does not require the strict replica ordering that
|
||
Chain Replication does. For any projection store key $K_n$, the
|
||
participating servers may have different values for $K_n$. As a
|
||
write-once store, it is impossible to mutate a replica of $K_n$. If
|
||
replicas of $K_n$ differ, then other parts of the system (projection
|
||
calculation and storage) are responsible for reconciling the
|
||
differences by writing a later key,
|
||
$K_{n+x}$ when $x>0$, with a new projection.
|
||
|
||
Projection store reads are ``best effort''. The projection used is chosen from
|
||
all replica servers that are available at the time of the read. The
|
||
minimum number of replicas is only one: the local projection store
|
||
should always be available, even if no other remote replica projection
|
||
stores are available.
|
||
|
||
For any key $K$, different projection stores $S_a$ and $S_b$ may store
|
||
nothing (i.e., {\tt error\_unwritten} when queried) or store different
|
||
values, $P_a \ne P_b$, despite having the same projection epoch
|
||
number. The following ranking rules are used to
|
||
determine the ``best value'' of a projection, where highest rank of
|
||
{\em any single projection} is considered the ``best value'':
|
||
|
||
\begin{enumerate}
|
||
\item An unwritten value is ranked at a value of $-1$.
|
||
\item A value whose {\tt author\_server} is at the $I^{th}$ position
|
||
in the {\tt all\_members} list has a rank of $I$.
|
||
\item A value whose {\tt dbg\_annotations} and/or other fields have
|
||
additional information may increase/decrease its rank, e.g.,
|
||
increase the rank by $10.25$.
|
||
\end{enumerate}
|
||
|
||
Rank rules \#2 and \#3 are intended to avoid worst-case ``thrashing''
|
||
of different projection proposals.
|
||
|
||
The concept of ``read repair'' of an unwritten key is the same as
|
||
Chain Replication's. If a read attempt for a key $K$ at some server
|
||
$S$ results in {\tt error\_unwritten}, then all of the other stores in
|
||
the {\tt \#projection.all\_members} list are consulted. If there is a
|
||
unanimous value $V_{u}$ elsewhere, then $V_{u}$ is use to repair all
|
||
unwritten replicas. If the value of $K$ is not unanimous, then the
|
||
``best value'' $V_{best}$ is used for the repair. If all respond with
|
||
{\tt error\_unwritten}, repair is not required.
|
||
|
||
\subsection{Adoption of new projections}
|
||
|
||
The projection store's ``best value'' for the largest written epoch
|
||
number at the time of the read is projection used by the FLU.
|
||
If the read attempt for projection $P_p$
|
||
also yields other non-best values, then the
|
||
projection calculation subsystem is notified. This notification
|
||
may/may not trigger a calculation of a new projection $P_{p+1}$ which
|
||
may eventually be stored and so
|
||
resolve $P_p$'s replicas' ambiguity.
|
||
|
||
\subsubsection{Alternative implementations: Hibari's ``Admin Server''
|
||
and Elastic Chain Replication}
|
||
|
||
See Section 7 of \cite{cr-theory-and-practice} for details of Hibari's
|
||
chain management agent, the ``Admin Server''. In brief:
|
||
|
||
\begin{itemize}
|
||
\item The Admin Server is intentionally a single point of failure in
|
||
the same way that the instance of Stanchion in a Riak CS cluster
|
||
is an intentional single
|
||
point of failure. In both cases, strict
|
||
serialization of state changes is more important than 100\%
|
||
availability.
|
||
|
||
\item For higher availability, the Hibari Admin Server is usually
|
||
configured in an active/standby manner. Status monitoring and
|
||
application failover logic is provided by the built-in capabilities
|
||
of the Erlang/OTP application controller.
|
||
|
||
\end{itemize}
|
||
|
||
Elastic chain replication is a technique described in
|
||
\cite{elastic-chain-replication}. It describes using multiple chains
|
||
to monitor each other, as arranged in a ring where a chain at position
|
||
$x$ is responsible for chain configuration and management of the chain
|
||
at position $x+1$. This technique is likely the fall-back to be used
|
||
in case the chain management method described in this RFC proves
|
||
infeasible.
|
||
|
||
\subsection{Likely problems and possible solutions}
|
||
\label{sub:likely-problems}
|
||
|
||
There are some unanswered questions about Machi's proposed chain
|
||
management technique. The problems that we guess are likely/possible
|
||
include:
|
||
|
||
\begin{itemize}
|
||
|
||
\item Thrashing or oscillating between a pair (or more) of
|
||
projections. It's hoped that the ``best projection'' ranking system
|
||
will be sufficient to prevent endless thrashing of projections, but
|
||
it isn't yet clear that it will be.
|
||
|
||
\item Partial (and/or one-way) network splits which cause partially
|
||
connected graphs of inter-node connectivity. Groups of nodes that
|
||
are completely isolated aren't a problem. However, partially
|
||
connected groups of nodes is an unknown. Intuition says that
|
||
communication (via the projection store) with ``bridge nodes'' in a
|
||
partially-connected network ought to settle eventually on a
|
||
projection with high rank, e.g., the projection on an island
|
||
subcluster of nodes with the largest author node name. Some corner
|
||
case(s) may exist where this intuition is not correct.
|
||
|
||
\item CP Mode management via the method proposed in
|
||
Section~\ref{sec:split-brain-management} may not be sufficient in
|
||
all cases.
|
||
|
||
\end{itemize}
|
||
|
||
\section{Chain Replication: proof of correctness}
|
||
\label{sub:cr-proof}
|
||
|
||
See Section~3 of \cite{chain-replication} for a proof of the
|
||
correctness of Chain Replication. A short summary is provide here.
|
||
Readers interested in good karma should read the entire paper.
|
||
|
||
The three basic rules of Chain Replication and its strong
|
||
consistency guarantee:
|
||
|
||
\begin{enumerate}
|
||
|
||
\item All replica servers are arranged in an ordered list $C$.
|
||
|
||
\item All mutations of a datum are performed upon each replica of $C$
|
||
strictly in the order which they appear in $C$. A mutation is considered
|
||
completely successful if the writes by all replicas are successful.
|
||
|
||
\item The head of the chain makes the determination of the order of
|
||
all mutations to all members of the chain. If the head determines
|
||
that some mutation $M_i$ happened before another mutation $M_j$,
|
||
then mutation $M_i$ happens before $M_j$ on all other members of
|
||
the chain.\footnote{While necesary for general Chain Replication,
|
||
Machi does not need this property. Instead, the property is
|
||
provided by Machi's sequencer and the write-once register of each
|
||
byte in each file.}
|
||
|
||
\item All read-only operations are performed by the ``tail'' replica,
|
||
i.e., the last replica in $C$.
|
||
|
||
\end{enumerate}
|
||
|
||
The basis of the proof lies in a simple logical trick, which is to
|
||
consider the history of all operations made to any server in the chain
|
||
as a literal list of unique symbols, one for each mutation.
|
||
|
||
Each replica of a datum will have a mutation history list. We will
|
||
call this history list $H$. For the $i^{th}$ replica in the chain list
|
||
$C$, we call $H_i$ the mutation history list for the $i^{th}$ replica.
|
||
|
||
Before the $i^{th}$ replica in the chain list begins service, its mutation
|
||
history $H_i$ is empty, $[]$. After this replica runs in a Chain
|
||
Replication system for a while, its mutation history list grows to
|
||
look something like
|
||
$[M_0, M_1, M_2, ..., M_{m-1}]$ where $m$ is the total number of
|
||
mutations of the datum that this server has processed successfully.
|
||
|
||
Let's assume for a moment that all mutation operations have stopped.
|
||
If the order of the chain was constant, and if all mutations are
|
||
applied to each replica in the chain's order, then all replicas of a
|
||
datum will have the exact same mutation history: $H_i = H_J$ for any
|
||
two replicas $i$ and $j$ in the chain
|
||
(i.e., $\forall i,j \in C, H_i = H_J$). That's a lovely property,
|
||
but it is much more interesting to assume that the service is
|
||
not stopped. Let's look next at a running system.
|
||
|
||
\begin{figure*}
|
||
\centering
|
||
\begin{tabular}{ccc}
|
||
{\bf {{On left side of $C$}}} & & {\bf On right side of $C$} \\
|
||
\hline
|
||
\multicolumn{3}{l}{Looking at replica order in chain $C$:} \\
|
||
$i$ & $<$ & $j$ \\
|
||
|
||
\multicolumn{3}{l}{For example:} \\
|
||
|
||
0 & $<$ & 2 \\
|
||
\hline
|
||
\multicolumn{3}{l}{It {\em must} be true: history lengths per replica:} \\
|
||
length($H_i$) & $\geq$ & length($H_j$) \\
|
||
\multicolumn{3}{l}{For example, a quiescent chain:} \\
|
||
48 & $\geq$ & 48 \\
|
||
\multicolumn{3}{l}{For example, a chain being mutated:} \\
|
||
55 & $\geq$ & 48 \\
|
||
\multicolumn{3}{l}{Example ordered mutation sets:} \\
|
||
$[M_0,M_1,\ldots,M_{46},M_{47},\ldots,M_{53},M_{54}]$ & $\supset$ & $[M_0,M_1,\ldots,M_{46},M_{47}]$ \\
|
||
\multicolumn{3}{c}{\bf Therefore the right side is always an ordered
|
||
subset} \\
|
||
\multicolumn{3}{c}{\bf of the left side. Furthermore, the ordered
|
||
sets on both} \\
|
||
\multicolumn{3}{c}{\bf sides have the exact same order of those elements they have in common.} \\
|
||
\multicolumn{3}{c}{The notation used by the Chain Replication paper is
|
||
shown below:} \\
|
||
$[M_0,M_1,\ldots,M_{46},M_{47},\ldots,M_{53},M_{54}]$ & $\succeq$ & $[M_0,M_1,\ldots,M_{46},M_{47}]$ \\
|
||
|
||
\end{tabular}
|
||
\caption{A demonstration of Chain Replication protocol history ``Update Propagation Invariant''.}
|
||
\label{tab:chain-order}
|
||
\end{figure*}
|
||
|
||
If the entire chain $C$ is processing any number of concurrent
|
||
mutations, then we can still understand $C$'s behavior.
|
||
Figure~\ref{tab:chain-order} shows us two replicas in chain $C$:
|
||
replica $R_i$ that's on the left/earlier side of the replica chain $C$
|
||
than some other replica $R_j$. We know that $i$'s position index in
|
||
the chain is smaller than $j$'s position index, so therefore $i < j$.
|
||
The restrictions of Chain Replication make it true that length($H_i$)
|
||
$\ge$ length($H_j$) because it's also that $H_i \supset H_j$, i.e,
|
||
$H_i$ on the left is always is a superset of $H_j$ on the right.
|
||
|
||
When considering $H_i$ and $H_j$ as strictly ordered lists, we have
|
||
$H_i \succeq H_j$, where the right side is always an exact prefix of the left
|
||
side's list. This prefixing propery is exactly what strong
|
||
consistency requires. If a value is read from the tail of the chain,
|
||
then no other chain member can have a prior/older value because their
|
||
respective mutations histories cannot be shorter than the tail
|
||
member's history.
|
||
|
||
\paragraph{``Update Propagation Invariant''}
|
||
is the original chain replication paper's name for the
|
||
$H_i \succeq H_j$
|
||
property. This paper will use the same name.
|
||
|
||
\section{Repair of entire files}
|
||
\label{sec:repair-entire-files}
|
||
|
||
There are some situations where repair of entire files is necessary.
|
||
|
||
\begin{itemize}
|
||
\item To repair FLUs added to a chain in a projection change,
|
||
specifically adding a new FLU to the chain. This case covers both
|
||
adding a new, data-less FLU and re-adding a previous, data-full FLU
|
||
back to the chain.
|
||
\item To avoid data loss when changing the order of the chain's servers.
|
||
\end{itemize}
|
||
|
||
Both situations can set the stage for data loss in the future.
|
||
If a violation of the Update Propagation Invariant (see end of
|
||
Section~\ref{sub:cr-proof}) is permitted, then the strong consistency
|
||
guarantee of Chain Replication is violated. Because Machi uses
|
||
write-once registers, the number of possible strong consistency
|
||
violations is small: any client that witnesses a written $\rightarrow$
|
||
unwritten transition is a violation of strong consistency. But
|
||
avoiding even this one bad scenario is a bit tricky.
|
||
|
||
As explained in Section~\ref{sub:data-loss1}, data
|
||
unavailability/loss when all chain servers fail is unavoidable. We
|
||
wish to avoid data loss whenever a chain has at least one surviving
|
||
server. Another method to avoid data loss is to preserve the Update
|
||
Propagation Invariant at all times.
|
||
|
||
\subsubsection{Just ``rsync'' it!}
|
||
\label{ssec:just-rsync-it}
|
||
|
||
A simple repair method might be perhaps 90\% sufficient.
|
||
That method could loosely be described as ``just {\tt rsync}
|
||
all files to all servers in an infinite loop.''\footnote{The
|
||
file format suggested in
|
||
\cite{machi-design} does not permit {\tt rsync}
|
||
as-is to be sufficient. A variation of {\tt rsync} would need to be
|
||
aware of the data/metadata split within each file and only replicate
|
||
the data section \ldots and the metadata would still need to be
|
||
managed outside of {\tt rsync}.}
|
||
|
||
However, such an informal method
|
||
cannot tell you exactly when you are in danger of data loss and when
|
||
data loss has actually happened. If we maintain the Update
|
||
Propagation Invariant, then we know exactly when data loss is immanent
|
||
or has happened.
|
||
|
||
Furthermore, we hope to use Machi for multiple use cases, including
|
||
ones that require strong consistency.
|
||
For uses such as CORFU, strong consistency is a non-negotiable
|
||
requirement. Therefore, we will use the Update Propagation Invariant
|
||
as the foundation for Machi's data loss prevention techniques.
|
||
|
||
\subsubsection{Divergence from CORFU: repair}
|
||
\label{sub:repair-divergence}
|
||
|
||
The original repair design for CORFU is simple and effective,
|
||
mostly. See Figure~\ref{fig:corfu-style-repair} for a full
|
||
description of the algorithm
|
||
Figure~\ref{fig:corfu-repair-sc-violation} for an example of a strong
|
||
consistency violation that can follow. (NOTE: This is a variation of
|
||
the data loss scenario that is described in
|
||
Figure~\ref{fig:data-loss2}.)
|
||
|
||
\begin{figure}
|
||
\begin{enumerate}
|
||
\item Destroy all data on the repair destination FLU.
|
||
\item Add the repair destination FLU to the tail of the chain in a new
|
||
projection $P_{p+1}$.
|
||
\item Change projection from $P_p$ to $P_{p+1}$.
|
||
\item Let single item read repair fix all of the problems.
|
||
\end{enumerate}
|
||
\caption{Simplest CORFU-style repair algorithm.}
|
||
\label{fig:corfu-style-repair}
|
||
\end{figure}
|
||
|
||
\begin{figure}
|
||
\begin{enumerate}
|
||
\item Write value $V$ to offset $O$ in the log with chain $[F_a]$.
|
||
This write is considered successful.
|
||
\item Change projection to configure chain as $[F_a,F_b]$. Prior to
|
||
the change, all values on FLU $F_b$ are unwritten.
|
||
\item FLU server $F_a$ crashes. The new projection defines the chain
|
||
as $[F_b]$.
|
||
\item A client attempts to read offset $O$ and finds an unwritten
|
||
value. This is a strong consistency violation.
|
||
%% \item The same client decides to fill $O$ with the junk value
|
||
%% $V_{junk}$. Now value $V$ is lost.
|
||
\end{enumerate}
|
||
\caption{An example scenario where the CORFU simplest repair algorithm
|
||
can lead to a violation of strong consistency.}
|
||
\label{fig:corfu-repair-sc-violation}
|
||
\end{figure}
|
||
|
||
A variation of the repair
|
||
algorithm is presented in section~2.5 of a later CORFU paper \cite{corfu2}.
|
||
However, the re-use a failed
|
||
server is not discussed there, either: the example of a failed server
|
||
$F_6$ uses a new server, $F_8$ to replace $F_6$. Furthermore, the
|
||
repair process is described as:
|
||
|
||
\begin{quote}
|
||
``Once $F_6$ is completely rebuilt on $F_8$ (by copying entries from
|
||
$F_7$), the system moves to projection (C), where $F_8$ is now used
|
||
to service all reads in the range $[40K,80K)$.''
|
||
\end{quote}
|
||
|
||
The phrase ``by copying entries'' does not give enough
|
||
detail to avoid the same data race as described in
|
||
Figure~\ref{fig:corfu-repair-sc-violation}. We believe that if
|
||
``copying entries'' means copying only written pages, then CORFU
|
||
remains vulnerable. If ``copying entries'' also means ``fill any
|
||
unwritten pages prior to copying them'', then perhaps the
|
||
vulnerability is eliminated.\footnote{SLF's note: Probably? This is my
|
||
gut feeling right now. However, given that I've just convinced
|
||
myself 100\% that fill during any possibility of split brain is {\em
|
||
not safe} in Machi, I'm not 100\% certain anymore than this ``easy''
|
||
fix for CORFU is correct.}.
|
||
|
||
\subsubsection{Whole-file repair as FLUs are (re-)added to a chain}
|
||
\label{sub:repair-add-to-chain}
|
||
|
||
Machi's repair process must preserve the Update Propagation
|
||
Invariant. To avoid data races with data copying from
|
||
``U.P.~Invariant preserving'' servers (i.e. fully repaired with
|
||
respect to the Update Propagation Invariant)
|
||
to servers of unreliable/unknown state, a
|
||
projection like the one shown in
|
||
Figure~\ref{fig:repair-chain-of-chains} is used. In addition, the
|
||
operations rules for data writes and reads must be observed in a
|
||
projection of this type.
|
||
|
||
\begin{figure*}
|
||
\centering
|
||
$
|
||
[\overbrace{\underbrace{H_1}_\textbf{Head of Heads}, M_{11},
|
||
\underbrace{T_1}_\textbf{Tail \#1}}^\textbf{Chain \#1 (U.P.~Invariant preserving)}
|
||
\mid
|
||
\overbrace{H_2, M_{21},
|
||
\underbrace{T_2}_\textbf{Tail \#2}}^\textbf{Chain \#2 (repairing)}
|
||
\mid \ldots \mid
|
||
\overbrace{H_n, M_{n1},
|
||
\underbrace{T_n}_\textbf{Tail \#n \& Tail of Tails ($T_{tails}$)}}^\textbf{Chain \#n (repairing)}
|
||
]
|
||
$
|
||
\caption{Representation of a ``chain of chains'': a chain prefix of
|
||
Update Propagation Invariant preserving FLUs (``Chain \#1'')
|
||
with FLUs from $n-1$ other chains under repair.}
|
||
\label{fig:repair-chain-of-chains}
|
||
\end{figure*}
|
||
|
||
\begin{itemize}
|
||
|
||
\item The system maintains the distinction between ``U.P.~preserving''
|
||
and ``repairing'' FLUs at all times. This allows the system to
|
||
track exactly which servers are known to preserve the Update
|
||
Propagation Invariant and which servers may/may not.
|
||
|
||
\item All ``repairing'' FLUs must be added only at the end of the
|
||
chain-of-chains.
|
||
|
||
\item All write operations must flow successfully through the
|
||
chain-of-chains from beginning to end, i.e., from the ``head of
|
||
heads'' to the ``tail of tails''. This rule also includes any
|
||
repair operations.
|
||
|
||
\item In AP Mode, all read operations are attempted from the list of
|
||
$[T_1,\-T_2,\-\ldots,\-T_n]$, where these FLUs are the tails of each of the
|
||
chains involved in repair.
|
||
In CP mode, all read operations are attempted only from $T_1$.
|
||
The first reply of {\tt \{ok, <<...>>\}} is a correct answer;
|
||
the rest of the FLU list can be ignored and the result returned to the
|
||
client. If all FLUs in the list have an unwritten value, then the
|
||
client can return {\tt error\_unwritten}.
|
||
|
||
\end{itemize}
|
||
|
||
While the normal single-write and single-read operations are performed
|
||
by the cluster, a file synchronization process is initiated. The
|
||
sequence of steps differs depending on the AP or CP mode of the system.
|
||
|
||
\paragraph{In cases where the cluster is operating in CP Mode:}
|
||
|
||
CORFU's repair method of ``just copy it all'' (from source FLU to repairing
|
||
FLU) is correct, {\em except} for the small problem pointed out in
|
||
Section~\ref{sub:repair-divergence}. The problem for Machi is one of
|
||
time \& space. Machi wishes to avoid transferring data that is
|
||
already correct on the repairing nodes. If a Machi node is storing
|
||
20TBytes of data, we really do not wish to use 20TBytes of bandwidth
|
||
to repair only 1 GByte of truly-out-of-sync data.
|
||
|
||
However, it is {\em vitally important} that all repairing FLU data be
|
||
clobbered/overwritten with exactly the same data as the Update
|
||
Propagation Invariant preserving chain. If this rule is not strictly
|
||
enforced, then fill operations can corrupt Machi file data. The
|
||
algorithm proposed is:
|
||
|
||
\begin{enumerate}
|
||
|
||
\item Change the projection to a ``chain of chains'' configuration
|
||
such as depicted in Figure~\ref{fig:repair-chain-of-chains}.
|
||
|
||
\item For all files on all FLUs in all chains, extract the lists of
|
||
written/unwritten byte ranges and their corresponding file data
|
||
checksums. (The checksum metadata is not strictly required for
|
||
recovery in AP Mode.)
|
||
Send these lists to the tail of tails
|
||
$T_{tails}$, which will collate all of the lists into a list of
|
||
tuples such as {\tt \{FName, $O_{start}, O_{end}$, CSum, FLU\_List\}}
|
||
where {\tt FLU\_List} is the list of all FLUs in the entire chain of
|
||
chains where the bytes at the location {\tt \{FName, $O_{start},
|
||
O_{end}$\}} are known to be written (as of the current repair period).
|
||
|
||
\item For chain \#1 members, i.e., the
|
||
leftmost chain relative to Figure~\ref{fig:repair-chain-of-chains},
|
||
repair files byte ranges for any chain \#1 members that are not members
|
||
of the {\tt FLU\_List} set. This will repair any partial
|
||
writes to chain \#1 that were unsuccessful (e.g., client crashed).
|
||
(Note however that this step only repairs FLUs in chain \#1.)
|
||
|
||
\item For all file byte ranges in all files on all FLUs in all
|
||
repairing chains where Tail \#1's value is unwritten, force all
|
||
repairing FLUs to also be unwritten.
|
||
|
||
\item For file byte ranges in all files on all FLUs in all repairing
|
||
chains where Tail \#1's value is written, send repair file byte data
|
||
\& metadata to any repairing FLU if the value repairing FLU's
|
||
value is unwritten or the checksum is not exactly equal to Tail \#1's
|
||
checksum.
|
||
|
||
\end{enumerate}
|
||
|
||
\begin{figure}
|
||
\centering
|
||
$
|
||
[\overbrace{\underbrace{H_1}_\textbf{Head}, M_{11}, T_1,
|
||
H_2, M_{21}, T_2,
|
||
\ldots
|
||
H_n, M_{n1},
|
||
\underbrace{T_n}_\textbf{Tail}}^\textbf{Chain (U.P.~Invariant preserving)}
|
||
]
|
||
$
|
||
\caption{Representation of Figure~\ref{fig:repair-chain-of-chains}
|
||
after all repairs have finished successfully and a new projection has
|
||
been calculated.}
|
||
\label{fig:repair-chain-of-chains-finished}
|
||
\end{figure}
|
||
|
||
When the repair is known to have copied all missing data successfully,
|
||
then the chain can change state via a new projection that includes the
|
||
repaired FLU(s) at the end of the U.P.~Invariant preserving chain \#1
|
||
in the same order in which they appeared in the chain-of-chains during
|
||
repair. See Figure~\ref{fig:repair-chain-of-chains-finished}.
|
||
|
||
The repair can be coordinated and/or performed by the $T_{tails}$ FLU
|
||
or any other FLU or cluster member that has spare capacity.
|
||
|
||
There is no serious race condition here between the enumeration steps
|
||
and the repair steps. Why? Because the change in projection at
|
||
step \#1 will force any new data writes to adapt to a new projection.
|
||
Consider the mutations that either happen before or after a projection
|
||
change:
|
||
|
||
|
||
\begin{itemize}
|
||
|
||
\item For all mutations $M_1$ prior to the projection change, the
|
||
enumeration steps \#3 \& \#4 and \#5 will always encounter mutation
|
||
$M_1$. Any repair must write through the entire chain-of-chains and
|
||
thus will preserve the Update Propagation Invariant when repair is
|
||
finished.
|
||
|
||
\item For all mutations $M_2$ starting during or after the projection
|
||
change has finished, a new mutation $M_2$ may or may not be included in the
|
||
enumeration steps \#3 \& \#4 and \#5.
|
||
However, in the new projection, $M_2$ must be
|
||
written to all chain of chains members, and such
|
||
in-order writes will also preserve the Update
|
||
Propagation Invariant and therefore is also be safe.
|
||
|
||
\end{itemize}
|
||
|
||
%% Then the only remaining safety problem (as far as I can see) is
|
||
%% avoiding this race:
|
||
|
||
%% \begin{enumerate}
|
||
%% \item Enumerate byte ranges $[B_0,B_1,\ldots]$ in file $F$ that must
|
||
%% be copied to the repair target, based on checksum differences for
|
||
%% those byte ranges.
|
||
%% \item A real-time concurrent write for byte range $B_x$ arrives at the
|
||
%% U.P.~Invariant preserving chain for file $F$ but was not a member of
|
||
%% step \#1's list of byte ranges.
|
||
%% \item Step \#2's update is propagated down the chain of chains.
|
||
%% \item Step \#1's clobber updates are propagated down the chain of
|
||
%% chains.
|
||
%% \item The value for $B_x$ is lost on the repair targets.
|
||
%% \end{enumerate}
|
||
|
||
\paragraph{In cases the cluster is operating in AP Mode:}
|
||
|
||
\begin{enumerate}
|
||
\item Follow the first two steps of the ``CP Mode''
|
||
sequence (above).
|
||
\item Follow step \#3 of the ``strongly consistent mode'' sequence
|
||
(above), but in place of repairing only FLUs in Chain \#1, AP mode
|
||
will repair the byte range of any FLU that is not a member of the
|
||
{\tt FLU\_List} set.
|
||
\item End of procedure.
|
||
\end{enumerate}
|
||
|
||
The end result is a huge ``merge'' where any
|
||
{\tt \{FName, $O_{start}, O_{end}$\}} range of bytes that is written
|
||
on FLU $F_w$ but missing/unwritten from FLU $F_m$ is written down the full chain
|
||
of chains, skipping any FLUs where the data is known to be written.
|
||
Such writes will also preserve Update Propagation Invariant when
|
||
repair is finished.
|
||
|
||
\subsubsection{Whole-file repair when changing FLU ordering within a chain}
|
||
\label{sub:repair-chain-re-ordering}
|
||
|
||
Changing FLU order within a chain is an operations optimization only.
|
||
It may be that the administrator wishes the order of a chain to remain
|
||
as originally configured during steady-state operation, e.g.,
|
||
$[F_a,F_b,F_c]$. As FLUs are stopped \& restarted, the chain may
|
||
become re-ordered in a seemingly-arbitrary manner.
|
||
|
||
It is certainly possible to re-order the chain, in a kludgy manner.
|
||
For example, if the desired order is $[F_a,F_b,F_c]$ but the current
|
||
operating order is $[F_c,F_b,F_a]$, then remove $F_b$ from the chain,
|
||
then add $F_b$ to the end of the chain. Then repeat the same
|
||
procedure for $F_c$. The end result will be the desired order.
|
||
|
||
From an operations perspective, re-ordering of the chain
|
||
using this kludgy manner has a
|
||
negative effect on availability: the chain is temporarily reduced from
|
||
operating with $N$ replicas down to $N-1$. This reduced replication
|
||
factor will not remain for long, at most a few minutes at a time, but
|
||
even a small amount of time may be unacceptable in some environments.
|
||
|
||
Reordering is possible with the introduction of a ``temporary head''
|
||
of the chain. This temporary FLU does not need to be a full replica
|
||
of the entire chain --- it merely needs to store replicas of mutations
|
||
that are made during the chain reordering process. This method will
|
||
not be described here. However, {\em if reviewers believe that it should
|
||
be included}, please let the authors know.
|
||
|
||
\paragraph{In both Machi operating modes:}
|
||
After initial implementation, it may be that the repair procedure is a
|
||
bit too slow. In order to accelerate repair decisions, it would be
|
||
helpful have a quicker method to calculate which files have exactly
|
||
the same contents. In traditional systems, this is done with a single
|
||
file checksum; see also the ``checksum scrub'' subsection in
|
||
\cite{machi-design}.
|
||
Machi's files can be written out-of-order from a file offset point of
|
||
view, which violates the order which the traditional method for
|
||
calculating a full-file hash. If we recall out-of-temporal-order
|
||
example in the ``Append-only files'' section of \cite{machi-design},
|
||
the traditional method cannot
|
||
continue calculating the file checksum at offset 2 until the byte at
|
||
file offset 1 is written.
|
||
|
||
It may be advantageous for each FLU to maintain for each file a
|
||
checksum of a canonical representation of the
|
||
{\tt \{$O_{start},O_{end},$ CSum\}} tuples that the FLU must already
|
||
maintain. Then for any two FLUs that claim to store a file $F$, if
|
||
both FLUs have the same hash of $F$'s written map + checksums, then
|
||
the copies of $F$ on both FLUs are the same.
|
||
|
||
\section{``Split brain'' management in CP Mode}
|
||
\label{sec:split-brain-management}
|
||
|
||
Split brain management is a thorny problem. The method presented here
|
||
is one based on pragmatics. If it doesn't work, there isn't a serious
|
||
worry, because Machi's first serious use case all require only AP Mode.
|
||
If we end up falling back to ``use Riak Ensemble'' or ``use ZooKeeper'',
|
||
then perhaps that's
|
||
fine enough. Meanwhile, let's explore how a
|
||
completely self-contained, no-external-dependencies
|
||
CP Mode Machi might work.
|
||
|
||
Wikipedia's description of the quorum consensus solution\footnote{See
|
||
{\tt http://en.wikipedia.org/wiki/Split-brain\_(computing)}.} is nice
|
||
and short:
|
||
|
||
\begin{quotation}
|
||
A typical approach, as described by Coulouris et al.,[4] is to use a
|
||
quorum-consensus approach. This allows the sub-partition with a
|
||
majority of the votes to remain available, while the remaining
|
||
sub-partitions should fall down to an auto-fencing mode.
|
||
\end{quotation}
|
||
|
||
This is the same basic technique that
|
||
both Riak Ensemble and ZooKeeper use. Machi's
|
||
extensive use of write-registers are a big advantage when implementing
|
||
this technique. Also very useful is the Machi ``wedge'' mechanism,
|
||
which can automatically implement the ``auto-fencing'' that the
|
||
technique requires. All Machi servers that can communicate with only
|
||
a minority of other servers will automatically ``wedge'' themselves
|
||
and refuse all requests for service until communication with the
|
||
majority can be re-established.
|
||
|
||
\subsection{The quorum: witness servers vs. full servers}
|
||
|
||
In any quorum-consensus system, at least $2f+1$ participants are
|
||
required to survive $f$ participant failures. Machi can implement a
|
||
technique of ``witness servers'' servers to bring the total cost
|
||
somewhere in the middle, between $2f+1$ and $f+1$, depending on your
|
||
point of view.
|
||
|
||
A ``witness server'' is one that participates in the network protocol
|
||
but does not store or manage all of the state that a ``full server''
|
||
does. A ``full server'' is a Machi server as
|
||
described by this RFC document. A ``witness server'' is a server that
|
||
only participates in the projection store and projection epoch
|
||
transition protocol and a small subset of the file access API.
|
||
A witness server doesn't actually store any
|
||
Machi files. A witness server is almost stateless, when compared to a
|
||
full Machi server.
|
||
|
||
A mixed cluster of witness and full servers must still contain at
|
||
least $2f+1$ participants. However, only $f+1$ of them are full
|
||
participants, and the remaining $f$ participants are witnesses. In
|
||
such a cluster, any majority quorum must have at least one full server
|
||
participant.
|
||
|
||
Witness FLUs are always placed at the front of the chain. As stated
|
||
above, there may be at most $f$ witness FLUs. A functioning quorum
|
||
majority
|
||
must have at least $f+1$ FLUs that can communicate and therefore
|
||
calculate and store a new unanimous projection. Therefore, any FLU at
|
||
the tail of a functioning quorum majority chain must be full FLU. Full FLUs
|
||
actually store Machi files, so they have no problem answering {\tt
|
||
read\_req} API requests.\footnote{We hope that it is now clear that
|
||
a witness FLU cannot answer any Machi file read API request.}
|
||
|
||
Any FLU that can only communicate with a minority of other FLUs will
|
||
find that none can calculate a new projection that includes a
|
||
majority of FLUs. Any such FLU, when in CP mode, would then move to
|
||
wedge state and remain wedged until the network partition heals enough
|
||
to communicate with the majority side. This is a nice property: we
|
||
automatically get ``fencing'' behavior.\footnote{Any FLU on the minority side
|
||
is wedged and therefore refuses to serve because it is, so to speak,
|
||
``on the wrong side of the fence.''}
|
||
|
||
There is one case where ``fencing'' may not happen: if both the client
|
||
and the tail FLU are on the same minority side of a network partition.
|
||
Assume the client and FLU $F_z$ are on the "wrong side" of a network
|
||
split; both are using projection epoch $P_1$. The tail of the
|
||
chain is $F_z$.
|
||
|
||
Also assume that the "right side" has reconfigured and is using
|
||
projection epoch $P_2$. The right side has mutated key $K$. Meanwhile,
|
||
nobody on the "right side" has noticed anything wrong and is happy to
|
||
continue using projection $P_1$.
|
||
|
||
\begin{itemize}
|
||
\item {\bf Option a}: Now the wrong side client reads $K$ using $P_1$ via
|
||
$F_z$. $F_z$ does not detect an epoch problem and thus returns an
|
||
answer. Given our assumptions, this value is stale. For some
|
||
client use cases, this kind of staleness may be OK in trade for
|
||
fewer network messages per read \ldots so Machi may
|
||
have a configurable option to permit it.
|
||
\item {\bf Option b}: The wrong side client must confirm that $P_1$ is
|
||
in use by a full majority of chain members, including $F_z$.
|
||
\end{itemize}
|
||
|
||
Attempts using Option b will fail for one of two reasons. First, if
|
||
the client can talk to a FLU that is using $P_2$, the client's
|
||
operation must be retried using $P_2$. Second, the client will time
|
||
out talking to enough FLUs so that it fails to get a quorum's worth of
|
||
$P_1$ answers. In either case, Option B will always fail a client
|
||
read and thus cannot return a stale value of $K$.
|
||
|
||
\subsection{Witness FLU data and protocol changes}
|
||
|
||
Some small changes to the projection's data structure
|
||
are required (relative to the initial spec described in
|
||
\cite{machi-design}). The projection itself
|
||
needs new annotation to indicate the operating mode, AP mode or CP
|
||
mode. The state type notifies the chain manager how to
|
||
react in network partitions and how to calculate new, safe projection
|
||
transitions and which file repair mode to use
|
||
(Section~\ref{sec:repair-entire-files}).
|
||
Also, we need to label member FLU servers as full- or
|
||
witness-type servers.
|
||
|
||
Write API requests are processed by witness servers in {\em almost but
|
||
not quite} no-op fashion. The only requirement of a witness server
|
||
is to return correct interpretations of local projection epoch
|
||
numbers, via the {\tt error\_bad\_epoch} and {\tt error\_wedged} error
|
||
codes. In fact, a new API call is sufficient for querying witness
|
||
servers: {\tt \{check\_epoch, m\_epoch()\}}.
|
||
Any client write operation sends the {\tt
|
||
check\_\-epoch} API command to witness FLUs and sends the usual {\tt
|
||
write\_\-req} command to full FLUs.
|
||
|
||
\section{The safety of projection epoch transitions}
|
||
\label{sec:safety-of-transitions}
|
||
|
||
Machi uses the projection epoch transition algorithm and
|
||
implementation from CORFU, which is believed to be safe. However,
|
||
CORFU assumes a single, external, strongly consistent projection
|
||
store. Further, CORFU assumes that new projections are calculated by
|
||
an oracle that the rest of the CORFU system agrees is the sole agent
|
||
for creating new projections. Such an assumption is impractical for
|
||
Machi's intended purpose.
|
||
|
||
Machi could use Riak Ensemble or ZooKeeper as an oracle (or perhaps as a oracle
|
||
coordinator), but we wish to keep Machi free of big external
|
||
dependencies. We would also like to see Machi be able to
|
||
operate in an ``AP mode'', which means providing service even
|
||
if all network communication to an oracle is broken.
|
||
|
||
The model of projection calculation and storage described in
|
||
Section~\ref{sec:projections} allows for each server to operate
|
||
independently, if necessary. This autonomy allows the server in AP
|
||
mode to
|
||
always accept new writes: new writes are written to unique file names
|
||
and unique file offsets using a chain consisting of only a single FLU,
|
||
if necessary. How is this possible? Let's look at a scenario in
|
||
Section~\ref{sub:split-brain-scenario}.
|
||
|
||
\subsection{A split brain scenario}
|
||
\label{sub:split-brain-scenario}
|
||
|
||
\begin{enumerate}
|
||
|
||
\item Assume 3 Machi FLUs, all in good health and perfect data sync: $[F_a,
|
||
F_b, F_c]$ using projection epoch $P_p$.
|
||
|
||
\item Assume data $D_0$ is written at offset $O_0$ in Machi file
|
||
$F_0$.
|
||
|
||
\item Then a network partition happens. Servers $F_a$ and $F_b$ are
|
||
on one side of the split, and server $F_c$ is on the other side of
|
||
the split. We'll call them the ``left side'' and ``right side'',
|
||
respectively.
|
||
|
||
\item On the left side, $F_b$ calculates a new projection and writes
|
||
it unanimously (to two projection stores) as epoch $P_B+1$. The
|
||
subscript $_B$ denotes a
|
||
version of projection epoch $P_{p+1}$ that was created by server $F_B$
|
||
and has a unique checksum (used to detect differences after the
|
||
network partition heals).
|
||
|
||
\item In parallel, on the right side, $F_c$ calculates a new
|
||
projection and writes it unanimously (to a single projection store)
|
||
as epoch $P_c+1$.
|
||
|
||
\item In parallel, a client on the left side writes data $D_1$
|
||
at offset $O_1$ in Machi file $F_1$, and also
|
||
a client on the right side writes data $D_2$
|
||
at offset $O_2$ in Machi file $F_2$. We know that $F_1 \ne F_2$
|
||
because each sequencer is forced to choose disjoint filenames from
|
||
any prior epoch whenever a new projection is available.
|
||
|
||
\end{enumerate}
|
||
|
||
Now, what happens when various clients attempt to read data values
|
||
$D_0$, $D_1$, and $D_2$?
|
||
|
||
\begin{itemize}
|
||
\item All clients can read $D_0$.
|
||
\item Clients on the left side can read $D_1$.
|
||
\item Attempts by clients on the right side to read $D_1$ will get
|
||
{\tt error\_unavailable}.
|
||
\item Clients on the right side can read $D_2$.
|
||
\item Attempts by clients on the left side to read $D_2$ will get
|
||
{\tt error\_unavailable}.
|
||
\end{itemize}
|
||
|
||
The {\tt error\_unavailable} result is not an error in the CAP Theorem
|
||
sense: it is a valid and affirmative response. In both cases, the
|
||
system on the client's side definitely knows that the cluster is
|
||
partitioned. If Machi were not a write-once store, perhaps there
|
||
might be an old/stale value to read on the local side of the network
|
||
partition \ldots but the system also knows definitely that no
|
||
old/stale value exists. Therefore Machi remains available in the
|
||
CAP Theorem sense both for writes and reads.
|
||
|
||
We know that all files $F_0$,
|
||
$F_1$, and $F_2$ are disjoint and can be merged (in a manner analogous
|
||
to set union) onto each server in $[F_a, F_b, F_c]$ safely
|
||
when the network partition is healed. However,
|
||
unlike pure theoretical set union, Machi's data merge \& repair
|
||
operations must operate within some constraints that are designed to
|
||
prevent data loss.
|
||
|
||
\subsection{Aside: defining data availability and data loss}
|
||
\label{sub:define-availability}
|
||
|
||
Let's take a moment to be clear about definitions:
|
||
|
||
\begin{itemize}
|
||
\item ``data is available at time $T$'' means that data is available
|
||
for reading at $T$: the Machi cluster knows for certain that the
|
||
requested data is not been written or it is written and has a single
|
||
value.
|
||
\item ``data is unavailable at time $T$'' means that data is
|
||
unavailable for reading at $T$ due to temporary circumstances,
|
||
e.g. network partition. If a read request is issued at some time
|
||
after $T$, the data will be available.
|
||
\item ``data is lost at time $T$'' means that data is permanently
|
||
unavailable at $T$ and also all times after $T$.
|
||
\end{itemize}
|
||
|
||
Chain Replication is a fantastic technique for managing the
|
||
consistency of data across a number of whole replicas. There are,
|
||
however, cases where CR can indeed lose data.
|
||
|
||
\subsection{Data loss scenario \#1: too few servers}
|
||
\label{sub:data-loss1}
|
||
|
||
If the chain is $N$ servers long, and if all $N$ servers fail, then
|
||
of course data is unavailable. However, if all $N$ fail
|
||
permanently, then data is lost.
|
||
|
||
If the administrator had intended to avoid data loss after $N$
|
||
failures, then the administrator would have provisioned a Machi
|
||
cluster with at least $N+1$ servers.
|
||
|
||
\subsection{Data Loss scenario \#2: bogus configuration change sequence}
|
||
\label{sub:data-loss2}
|
||
|
||
Assume that the sequence of events in Figure~\ref{fig:data-loss2} takes place.
|
||
|
||
\begin{figure}
|
||
\begin{enumerate}
|
||
%% NOTE: the following list 9 items long. We use that fact later, see
|
||
%% string YYY9 in a comment further below. If the length of this list
|
||
%% changes, then the counter reset below needs adjustment.
|
||
\item Projection $P_p$ says that chain membership is $[F_a]$.
|
||
\item A write of data $D$ to file $F$ at offset $O$ is successful.
|
||
\item Projection $P_{p+1}$ says that chain membership is $[F_a,F_b]$, via
|
||
an administration API request.
|
||
\item Machi will trigger repair operations, copying any missing data
|
||
files from FLU $F_a$ to FLU $F_b$. For the purpose of this
|
||
example, the sync operation for file $F$'s data and metadata has
|
||
not yet started.
|
||
\item FLU $F_a$ crashes.
|
||
\item The chain manager on $F_b$ notices $F_a$'s crash,
|
||
decides to create a new projection $P_{p+2}$ where chain membership is
|
||
$[F_b]$
|
||
successfully stores $P_{p+2}$ in its local store. FLU $F_b$ is now wedged.
|
||
\item FLU $F_a$ is down, therefore the
|
||
value of $P_{p+2}$ is unanimous for all currently available FLUs
|
||
(namely $[F_b]$).
|
||
\item FLU $F_b$ sees that projection $P_{p+2}$ is the newest unanimous
|
||
projection. It unwedges itself and continues operation using $P_{p+2}$.
|
||
\item Data $D$ is definitely unavailable for now, perhaps lost forever?
|
||
\end{enumerate}
|
||
\caption{Data unavailability scenario with danger of permanent data loss}
|
||
\label{fig:data-loss2}
|
||
\end{figure}
|
||
|
||
At this point, the data $D$ is not available on $F_b$. However, if
|
||
we assume that $F_a$ eventually returns to service, and Machi
|
||
correctly acts to repair all data within its chain, then $D$
|
||
all of its contents will be available eventually.
|
||
|
||
However, if server $F_a$ never returns to service, then $D$ is lost. The
|
||
Machi administration API must always warn the user that data loss is
|
||
possible. In Figure~\ref{fig:data-loss2}'s scenario, the API must
|
||
warn the administrator in multiple ways that fewer than the full {\tt
|
||
length(all\_members)} number of replicas are in full sync.
|
||
|
||
A careful reader should note that $D$ is also lost if step \#5 were
|
||
instead, ``The hardware that runs FLU $F_a$ was destroyed by fire.''
|
||
For any possible step following \#5, $D$ is lost. This is data loss
|
||
for the same reason that the scenario of Section~\ref{sub:data-loss1}
|
||
happens: the administrator has not provisioned a sufficient number of
|
||
replicas.
|
||
|
||
Let's revisit Figure~\ref{fig:data-loss2}'s scenario yet again. This
|
||
time, we add a final step at the end of the sequence:
|
||
|
||
\begin{enumerate}
|
||
\setcounter{enumi}{9} % YYY9
|
||
\item The administration API is used to change the chain
|
||
configuration to {\tt all\_members=$[F_b]$}.
|
||
\end{enumerate}
|
||
|
||
Step \#10 causes data loss. Specifically, the only copy of file
|
||
$F$ is on FLU $F_a$. By administration policy, FLU $F_a$ is now
|
||
permanently inaccessible.
|
||
|
||
The chain manager {\em must} keep track of all
|
||
repair operations and their status. If such information is tracked by
|
||
all FLUs, then the data loss by bogus administrator action can be
|
||
prevented. In this scenario, FLU $F_b$ knows that `$F_a \rightarrow
|
||
F_b$` repair has not yet finished and therefore it is unsafe to remove
|
||
$F_a$ from the cluster.
|
||
|
||
\subsection{Data Loss scenario \#3: chain replication repair done badly}
|
||
\label{sub:data-loss3}
|
||
|
||
It's quite possible to lose data through careless/buggy Chain
|
||
Replication chain configuration changes. For example, in the split
|
||
brain scenario of Section~\ref{sub:split-brain-scenario}, we have two
|
||
pieces of data written to different ``sides'' of the split brain,
|
||
$D_0$ and $D_1$. If the chain is naively reconfigured after the network
|
||
partition heals to be $[F_a=\emptyset,F_b=\emptyset,F_c=D_1],$\footnote{Where $\emptyset$
|
||
denotes the unwritten value.} then $D_1$
|
||
is in danger of being lost. Why?
|
||
The Update Propagation Invariant is violated.
|
||
Any Chain Replication read will be
|
||
directed to the tail, $F_c$. The value exists there, so there is no
|
||
need to do any further work; the unwritten values at $F_a$ and $F_b$
|
||
will not be repaired. If the $F_c$ server fails sometime
|
||
later, then $D_1$ will be lost. The ``Chain Replication Repair''
|
||
section of \cite{machi-design} discusses
|
||
how data loss can be avoided after servers are added (or re-added) to
|
||
an active chain configuration.
|
||
|
||
\subsection{Summary}
|
||
|
||
We believe that maintaining the Update Propagation Invariant is a
|
||
hassle anda pain, but that hassle and pain are well worth the
|
||
sacrifices required to maintain the invariant at all times. It avoids
|
||
data loss in all cases where the U.P.~Invariant preserving chain
|
||
contains at least one FLU.
|
||
|
||
\bibliographystyle{abbrvnat}
|
||
\begin{thebibliography}{}
|
||
\softraggedright
|
||
|
||
\bibitem{elastic-chain-replication}
|
||
Abu-Libdeh, Hussam et al.
|
||
Leveraging Sharding in the Design of Scalable Replication Protocols.
|
||
Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC'13), 2013.
|
||
{\tt http://www.ymsir.com/papers/sharding-socc.pdf}
|
||
|
||
\bibitem{corfu1}
|
||
Balakrishnan, Mahesh et al.
|
||
CORFU: A Shared Log Design for Flash Clusters.
|
||
Proceedings of the 9th USENIX Conference on Networked Systems Design
|
||
and Implementation (NSDI'12), 2012.
|
||
{\tt http://research.microsoft.com/pubs/157204/ corfumain-final.pdf}
|
||
|
||
\bibitem{corfu2}
|
||
Balakrishnan, Mahesh et al.
|
||
CORFU: A Distributed Shared Log
|
||
ACM Transactions on Computer Systems, Vol. 31, No. 4, Article 10, December 2013.
|
||
{\tt http://www.snookles.com/scottmp/corfu/ corfu.a10-balakrishnan.pdf}
|
||
|
||
\bibitem{machi-design}
|
||
Basho Japan KK.
|
||
Machi: an immutable file store
|
||
{\tt https://github.com/basho/machi/tree/ master/doc/high-level-machi.pdf}
|
||
|
||
\bibitem{was}
|
||
Calder, Brad et al.
|
||
Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency
|
||
Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP'11), 2011.
|
||
{\tt http://sigops.org/sosp/sosp11/current/ 2011-Cascais/printable/11-calder.pdf}
|
||
|
||
\bibitem{cr-theory-and-practice}
|
||
Fritchie, Scott Lystig.
|
||
Chain Replication in Theory and in Practice.
|
||
Proceedings of the 9th ACM SIGPLAN Workshop on Erlang (Erlang'10), 2010.
|
||
{\tt http://www.snookles.com/scott/publications/ erlang2010-slf.pdf}
|
||
|
||
\bibitem{the-log-what}
|
||
Kreps, Jay.
|
||
The Log: What every software engineer should know about real-time data's unifying abstraction
|
||
{\tt http://engineering.linkedin.com/distributed-
|
||
systems/log-what-every-software-engineer-should-
|
||
know-about-real-time-datas-unifying}
|
||
|
||
\bibitem{kafka}
|
||
Kreps, Jay et al.
|
||
Kafka: a distributed messaging system for log processing.
|
||
NetDB’11.
|
||
{\tt http://research.microsoft.com/en-us/UM/people/
|
||
srikanth/netdb11/netdb11papers/netdb11-final12.pdf}
|
||
|
||
\bibitem{paxos-made-simple}
|
||
Lamport, Leslie.
|
||
Paxos Made Simple.
|
||
In SIGACT News \#4, Dec, 2001.
|
||
{\tt http://research.microsoft.com/users/ lamport/pubs/paxos-simple.pdf}
|
||
|
||
\bibitem{random-slicing}
|
||
Miranda, Alberto et al.
|
||
Random Slicing: Efficient and Scalable Data Placement for Large-Scale Storage Systems.
|
||
ACM Transactions on Storage, Vol. 10, No. 3, Article 9, July 2014.
|
||
{\tt http://www.snookles.com/scottmp/corfu/random- slicing.a9-miranda.pdf}
|
||
|
||
\bibitem{porcupine}
|
||
Saito, Yasushi et al.
|
||
Manageability, availability and performance in Porcupine: a highly scalable, cluster-based mail service.
|
||
7th ACM Symposium on Operating System Principles (SOSP’99).
|
||
{\tt http://homes.cs.washington.edu/\%7Elevy/ porcupine.pdf}
|
||
|
||
\bibitem{chain-replication}
|
||
van Renesse, Robbert et al.
|
||
Chain Replication for Supporting High Throughput and Availability.
|
||
Proceedings of the 6th Conference on Symposium on Operating Systems
|
||
Design \& Implementation (OSDI'04) - Volume 6, 2004.
|
||
{\tt http://www.cs.cornell.edu/home/rvr/papers/ osdi04.pdf}
|
||
|
||
\end{thebibliography}
|
||
|
||
|
||
\end{document}
|