Doc update, including mid-December 2015 status #54
2 changed files with 72 additions and 84 deletions
119
README.md
119
README.md
|
|
@ -2,6 +2,16 @@
|
|||
|
||||
[Travis-CI](http://travis-ci.org/basho/machi) :: 
|
||||
|
||||
Outline
|
||||
|
||||
1. [Why another file store?](#sec1)
|
||||
2. [Development status summary](#sec2)
|
||||
3. [Where to learn more about Machi](#sec3)
|
||||
4. [Contributing to Machi's development](#sec4)
|
||||
|
||||
<a name="sec1">
|
||||
## 1. Why another file store?
|
||||
|
||||
Our goal is a robust & reliable, distributed, highly available, large
|
||||
file store. Such stores already exist, both in the open source world
|
||||
and in the commercial world. Why reinvent the wheel? We believe
|
||||
|
|
@ -38,40 +48,51 @@ corruption detection should be ubiquitous today. Alas, they are not.
|
|||
Machi is an effort to change the deplorable state of the world, one
|
||||
Erlang function at a time.
|
||||
|
||||
## Status: mid-December 2015: work is underway
|
||||
<a name="sec2">
|
||||
## 2. Development status summary
|
||||
|
||||
TODO: status update here.
|
||||
Mid-December 2015: work is underway.
|
||||
|
||||
* The chain manager is ready for both eventual consistency use ("available
|
||||
mode") and strong consistency use ("consistent mode"). Both modes use a new
|
||||
consensus technique, Humming Consensus.
|
||||
* Scott will be
|
||||
[speaking about Humming Consensus](http://ricon.io/agenda/#managing-chain-replication-metadata-with-humming-consensus)
|
||||
at the [Ricon 2015 conference] (http://ricon.io) in San Francisco,
|
||||
CA, USA on Thursday, November 5th, 2015.
|
||||
* If you would like to run the network partition simulator
|
||||
mentioned in that Ricon presentation, please see the
|
||||
[partition simulator convergence test doc.](./doc/machi_chain_manager1_converge_demo.md)
|
||||
* Implementation of the file repair process for strong consistency
|
||||
is still in progress.
|
||||
* In progress:
|
||||
* Code refactoring: metadata management using
|
||||
[ELevelDB](https://github.com/basho/eleveldb)
|
||||
* File repair using file-centric, Merkle-style hash tree.
|
||||
* QuickCheck tests for file repair correctness
|
||||
* 2015-12-15: The EUnit test `machi_ap_repair_eqc` is
|
||||
currently failing occasionally because it (correctly) detects
|
||||
double-write errors. Double-write errors will be eliminated
|
||||
when the ELevelDB integration work is complete.
|
||||
* Chain Replication management using the Humming Consensus
|
||||
algorithm to manage chain state is stable.
|
||||
* ... with the caveat that it runs very well in a very harsh
|
||||
and unforgiving network partition simulator but has not run
|
||||
much yet in the real world.
|
||||
* All Machi client/server protocols are based on
|
||||
[Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview).
|
||||
* The current specification for Machi's protocols can be found at
|
||||
[https://github.com/basho/machi/blob/master/src/machi.proto](https://github.com/basho/machi/blob/master/src/machi.proto).
|
||||
* The Machi PB protocol is not yet stable. Expect change!
|
||||
* The Erlang language client implementation of the high-level
|
||||
protocol flavor is brittle (e.g., little error handling yet).
|
||||
|
||||
* All Machi client/server protocols are based on
|
||||
[Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview).
|
||||
* The current specification for Machi's protocols can be found at
|
||||
[https://github.com/basho/machi/blob/master/src/machi.proto](https://github.com/basho/machi/blob/master/src/machi.proto).
|
||||
* The Machi PB protocol is not yet stable. Expect change!
|
||||
* The Erlang language client implementation of the high-level
|
||||
protocol flavor is brittle (e.g., little error handling yet).
|
||||
If you would like to run the network partition simulator
|
||||
mentioned in the Ricon 2015 presentation about Humming Consensus,
|
||||
please see the
|
||||
[partition simulator convergence test doc.](./doc/machi_chain_manager1_converge_demo.md)
|
||||
|
||||
If you'd like to work on a protocol such as Thrift, UBF,
|
||||
msgpack over UDP, or some other protocol, let us know by
|
||||
[opening an issue to discuss it](./issues/new).
|
||||
|
||||
## Where to learn more about Machi
|
||||
<a name="sec3">
|
||||
## 3. Where to learn more about Machi
|
||||
|
||||
The two major design documents for Machi are now mostly stable.
|
||||
Please see the [doc](./doc) directory's [README](./doc) for details.
|
||||
|
||||
We also have a
|
||||
[Frequently Asked Questions (FAQ) list](./FAQ.md).
|
||||
|
||||
Scott recently (November 2015) gave a presentation at the
|
||||
[RICON 2015 conference](http://ricon.io) about one of the techniques
|
||||
used by Machi; "Managing Chain Replication Metadata with
|
||||
|
|
@ -79,7 +100,10 @@ Humming Consensus" is available online now.
|
|||
* [slides (PDF format)](http://ricon.io/speakers/slides/Scott_Fritchie_Ricon_2015.pdf)
|
||||
* [video](https://www.youtube.com/watch?v=yR5kHL1bu1Q)
|
||||
|
||||
## Contributing to Machi: source code, documentation, etc.
|
||||
<a name="sec4">
|
||||
## 4. Contributing to Machi's development
|
||||
|
||||
### 4.1 License
|
||||
|
||||
Basho Technologies, Inc. as committed to licensing all work for Machi
|
||||
under the
|
||||
|
|
@ -95,26 +119,7 @@ We invite all contributors to review the
|
|||
[CONTRIBUTING.md](./CONTRIBUTING.md) document for guidelines for
|
||||
working with the Basho development team.
|
||||
|
||||
## A brief survey of this directories in this repository
|
||||
|
||||
* A list of Frequently Asked Questions, a.k.a.
|
||||
[the Machi FAQ](./FAQ.md).
|
||||
|
||||
* The [doc](./doc/) directory: home for major documents about Machi:
|
||||
high level design documents as well as exploration of features still
|
||||
under design & review within Basho.
|
||||
|
||||
* The `ebin` directory: used for compiled application code
|
||||
|
||||
* The `include`, `src`, and `test` directories: contain the header
|
||||
files, source files, and test code for Machi, respectively.
|
||||
|
||||
* The [prototype](./prototype/) directory: contains proof of concept
|
||||
code, scaffolding libraries, and other exploratory code. Curious
|
||||
readers should see the [prototype/README.md](./prototype/README.md)
|
||||
file for more explanation of the small sub-projects found here.
|
||||
|
||||
## Development environment requirements
|
||||
### 4.2 Development environment requirements
|
||||
|
||||
All development to date has been done with Erlang/OTP version 17 on OS
|
||||
X. The only known limitations for using R16 are minor type
|
||||
|
|
@ -126,26 +131,8 @@ tool chain for C and C++ applications. Specifically, we assume `make`
|
|||
is available. The utility used to compile the Machi source code,
|
||||
`rebar`, is pre-compiled and included in the repo.
|
||||
|
||||
There are no known OS limits at this time: any platform that supports
|
||||
Erlang/OTP should be sufficient for Machi. This may change over time
|
||||
(e.g., adding NIFs which can make full portability to Windows OTP
|
||||
environments difficult), but it hasn't happened yet.
|
||||
|
||||
## Contributions
|
||||
|
||||
Basho encourages contributions to Riak from the community. Here’s how
|
||||
to get started.
|
||||
|
||||
* Fork the appropriate sub-projects that are affected by your change.
|
||||
* Create a topic branch for your change and checkout that branch.
|
||||
git checkout -b some-topic-branch
|
||||
* Make your changes and run the test suite if one is provided. (see below)
|
||||
* Commit your changes and push them to your fork.
|
||||
* Open pull-requests for the appropriate projects.
|
||||
* Contributors will review your pull request, suggest changes, and merge it when it’s ready and/or offer feedback.
|
||||
* To report a bug or issue, please open a new issue against this repository.
|
||||
|
||||
-The Machi team at Basho,
|
||||
[Scott Lystig Fritchie](mailto:scott@basho.com), technical lead, and
|
||||
[Matt Brender](mailto:mbrender@basho.com), your developer advocate.
|
||||
|
||||
Machi has a dependency on the
|
||||
[ELevelDB](https://github.com/basho/eleveldb) library. ELevelDB only
|
||||
supports UNIX/Linux OSes and 64-bit versions of Erlang/OTP only; we
|
||||
apologize to Windows-based and 32-bit-based Erlang developers for this
|
||||
restriction.
|
||||
|
|
|
|||
|
|
@ -6,20 +6,6 @@ Erlang documentation, please use this link:
|
|||
|
||||
## Documents in this directory
|
||||
|
||||
### chain-self-management-sketch.org
|
||||
|
||||
[chain-self-management-sketch.org](chain-self-management-sketch.org)
|
||||
is a mostly-deprecated draft of
|
||||
an introduction to the
|
||||
self-management algorithm proposed for Machi. Most material has been
|
||||
moved to the [high-level-chain-mgr.pdf](high-level-chain-mgr.pdf) document.
|
||||
|
||||
### cluster-of-clusters (directory)
|
||||
|
||||
This directory contains the sketch of the "cluster of clusters" design
|
||||
strawman for partitioning/distributing/sharding files across a large
|
||||
number of independent Machi clusters.
|
||||
|
||||
### high-level-machi.pdf
|
||||
|
||||
[high-level-machi.pdf](high-level-machi.pdf)
|
||||
|
|
@ -50,9 +36,9 @@ introduction to the Humming Consensus algorithm. Its abstract:
|
|||
> of file updates to all replica servers in a Machi cluster. Chain
|
||||
> Replication is a variation of primary/backup replication where the
|
||||
> order of updates between the primary server and each of the backup
|
||||
> servers is strictly ordered into a single ``chain''. Management of
|
||||
> Chain Replication's metadata, e.g., ``What is the current order of
|
||||
> servers in the chain?'', remains an open research problem. The
|
||||
> servers is strictly ordered into a single "chain". Management of
|
||||
> Chain Replication's metadata, e.g., "What is the current order of
|
||||
> servers in the chain?", remains an open research problem. The
|
||||
> current state of the art for Chain Replication metadata management
|
||||
> relies on an external oracle (e.g., ZooKeeper) or the Elastic
|
||||
> Replication algorithm.
|
||||
|
|
@ -60,7 +46,7 @@ introduction to the Humming Consensus algorithm. Its abstract:
|
|||
> This document describes the Machi chain manager, the component
|
||||
> responsible for managing Chain Replication metadata state. The chain
|
||||
> manager uses a new technique, based on a variation of CORFU, called
|
||||
> ``humming consensus''.
|
||||
> "humming consensus".
|
||||
> Humming consensus does not require active participation by all or even
|
||||
> a majority of participants to make decisions. Machi's chain manager
|
||||
> bases its logic on humming consensus to make decisions about how to
|
||||
|
|
@ -71,3 +57,18 @@ introduction to the Humming Consensus algorithm. Its abstract:
|
|||
> decision during that epoch. When a differing decision is discovered,
|
||||
> new time epochs are proposed in which a new consensus is reached and
|
||||
> disseminated to all available participants.
|
||||
|
||||
### chain-self-management-sketch.org
|
||||
|
||||
[chain-self-management-sketch.org](chain-self-management-sketch.org)
|
||||
is a mostly-deprecated draft of
|
||||
an introduction to the
|
||||
self-management algorithm proposed for Machi. Most material has been
|
||||
moved to the [high-level-chain-mgr.pdf](high-level-chain-mgr.pdf) document.
|
||||
|
||||
### cluster-of-clusters (directory)
|
||||
|
||||
This directory contains the sketch of the "cluster of clusters" design
|
||||
strawman for partitioning/distributing/sharding files across a large
|
||||
number of independent Machi clusters.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue