Compare commits
No commits in common. "master" and "gh-pages" have entirely different histories.
298 changed files with 5726 additions and 77955 deletions
33
.gitignore
vendored
33
.gitignore
vendored
|
|
@ -1,33 +0,0 @@
|
|||
prototype/chain-manager/patch.*
|
||||
.eqc-info
|
||||
.eunit
|
||||
deps
|
||||
dev
|
||||
erl_crash.dump
|
||||
eqc
|
||||
.concrete/DEV_MODE
|
||||
.rebar
|
||||
edoc
|
||||
|
||||
# Dialyzer stuff
|
||||
.dialyzer-last-run.txt
|
||||
.ebin.native
|
||||
.local_dialyzer_plt
|
||||
dialyzer_unhandled_warnings
|
||||
dialyzer_warnings
|
||||
*.plt
|
||||
|
||||
# PB artifacts for Erlang
|
||||
include/machi_pb.hrl
|
||||
|
||||
# Release packaging
|
||||
rel/machi
|
||||
rel/vars/dev*vars.config
|
||||
|
||||
# Misc Scott cruft
|
||||
*.patch
|
||||
current_counterexample.eqc
|
||||
foo*
|
||||
RUNLOG*
|
||||
typescript*
|
||||
*.swp
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
language: erlang
|
||||
notifications:
|
||||
email: scott@basho.com
|
||||
script: "priv/test-for-gh-pr.sh"
|
||||
otp_release:
|
||||
- 17.5
|
||||
## No, Dialyzer is too different between 17 & 18: - 18.1
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
# Contributing to the Machi project
|
||||
|
||||
The most helpful way to contribute is by reporting your experience
|
||||
through issues. Issues may not be updated while we review internally,
|
||||
but they're still incredibly appreciated.
|
||||
|
||||
Pull requests may take multiple engineers for verification and testing. If
|
||||
you're passionate enough to want to learn more on how you can get
|
||||
hands on in this process, reach out to
|
||||
[Matt Brender](mailto:mbrender@basho.com), your developer advocate.
|
||||
|
||||
Thank you for being part of the community! We love you for it.
|
||||
|
||||
## If you have a question or wish to provide design feedback/criticism
|
||||
|
||||
Please
|
||||
[open a support ticket at GitHub](https://github.com/basho/machi/issues/new)
|
||||
to ask questions and to provide feedback about Machi's
|
||||
design/documentation/source code.
|
||||
|
||||
## General development process
|
||||
|
||||
Machi is still a very young project within Basho, with a small team of
|
||||
developers; please bear with us as we grow out of "toddler" stage into
|
||||
a more mature open source software project.
|
||||
|
||||
* Fork the Machi source repo and/or the sub-projects that are affected
|
||||
by your change.
|
||||
* Create a topic branch for your change and checkout that branch.
|
||||
git checkout -b some-topic-branch
|
||||
* Make your changes and run the test suite if one is provided.
|
||||
* Commit your changes and push them to your fork.
|
||||
* Open pull-requests for the appropriate projects.
|
||||
* Contributors will review your pull request, suggest changes, and merge it when it’s ready and/or offer feedback.
|
||||
* To report a bug or issue, please open a new issue against this repository.
|
||||
630
FAQ.md
630
FAQ.md
|
|
@ -1,630 +0,0 @@
|
|||
# Frequently Asked Questions (FAQ)
|
||||
|
||||
<!-- Formatting: -->
|
||||
<!-- All headings omitted from outline are H1 -->
|
||||
<!-- All other headings must be on a single line! -->
|
||||
<!-- Run: ./priv/make-faq.pl ./FAQ.md > ./tmpfoo; mv ./tmpfoo ./FAQ.md -->
|
||||
|
||||
# Outline
|
||||
|
||||
<!-- OUTLINE -->
|
||||
|
||||
+ [1 Questions about Machi in general](#n1)
|
||||
+ [1.1 What is Machi?](#n1.1)
|
||||
+ [1.2 What is a Machi chain?](#n1.2)
|
||||
+ [1.3 What is a Machi cluster?](#n1.3)
|
||||
+ [1.4 What is Machi like when operating in "eventually consistent" mode?](#n1.4)
|
||||
+ [1.5 What is Machi like when operating in "strongly consistent" mode?](#n1.5)
|
||||
+ [1.6 What does Machi's API look like?](#n1.6)
|
||||
+ [1.7 What licensing terms are used by Machi?](#n1.7)
|
||||
+ [1.8 Where can I find the Machi source code and documentation? Can I contribute?](#n1.8)
|
||||
+ [1.9 What is Machi's expected release schedule, packaging, and operating system/OS distribution support?](#n1.9)
|
||||
+ [2 Questions about Machi relative to {{something else}}](#n2)
|
||||
+ [2.1 How is Machi better than Hadoop?](#n2.1)
|
||||
+ [2.2 How does Machi differ from HadoopFS/HDFS?](#n2.2)
|
||||
+ [2.3 How does Machi differ from Kafka?](#n2.3)
|
||||
+ [2.4 How does Machi differ from Bookkeeper?](#n2.4)
|
||||
+ [2.5 How does Machi differ from CORFU and Tango?](#n2.5)
|
||||
+ [3 Machi's specifics](#n3)
|
||||
+ [3.1 What technique is used to replicate Machi's files? Can other techniques be used?](#n3.1)
|
||||
+ [3.2 Does Machi have a reliance on a coordination service such as ZooKeeper or etcd?](#n3.2)
|
||||
+ [3.3 Are there any presentations available about Humming Consensus](#n3.3)
|
||||
+ [3.4 Is it true that there's an allegory written to describe Humming Consensus?](#n3.4)
|
||||
+ [3.5 How is Machi tested?](#n3.5)
|
||||
+ [3.6 Does Machi require shared disk storage? e.g. iSCSI, NBD (Network Block Device), Fibre Channel disks](#n3.6)
|
||||
+ [3.7 Does Machi require or assume that servers with large numbers of disks must use RAID-0/1/5/6/10/50/60 to create a single block device?](#n3.7)
|
||||
+ [3.8 What language(s) is Machi written in?](#n3.8)
|
||||
+ [3.9 Can Machi run on Windows? Can Machi run on 32-bit platforms?](#n3.9)
|
||||
+ [3.10 Does Machi use the Erlang/OTP network distribution system (aka "disterl")?](#n3.10)
|
||||
+ [3.11 Can I use HTTP to write/read stuff into/from Machi?](#n3.11)
|
||||
|
||||
<!-- ENDOUTLINE -->
|
||||
|
||||
<a name="n1">
|
||||
## 1. Questions about Machi in general
|
||||
|
||||
<a name="n1.1">
|
||||
### 1.1. What is Machi?
|
||||
|
||||
Very briefly, Machi is a very simple append-only blob/file store.
|
||||
|
||||
Machi is
|
||||
"dumber" than many other file stores (i.e., lacking many features
|
||||
found in other file stores) such as HadoopFS or a simple NFS or CIFS file
|
||||
server.
|
||||
However, Machi is a distributed blob/file store, which makes it different
|
||||
(and, in some ways, more complicated) than a simple NFS or CIFS file
|
||||
server.
|
||||
|
||||
All Machi data is protected by SHA-1 checksums. By default, these
|
||||
checksums are calculated by the client to provide strong end-to-end
|
||||
protection against data corruption. (If the client does not provide a
|
||||
checksum, one will be generated by the first Machi server to handle
|
||||
the write request.) Internally, Machi uses these checksums for local
|
||||
data integrity checks and for server-to-server file synchronization
|
||||
and corrupt data repair.
|
||||
|
||||
As a distributed system, Machi can be configured to operate with
|
||||
either eventually consistent mode or strongly consistent mode. In
|
||||
strongly consistent mode, Machi can provide write-once file store
|
||||
service in the same style as CORFU. Machi can be an easy to use tool
|
||||
for building fully ordered, log-based distributed systems and
|
||||
distributed data structures.
|
||||
|
||||
In eventually consistent mode, Machi can remain available for writes
|
||||
during arbitrary network partitions. When a network partition is
|
||||
fixed, Machi can safely merge all file data together without data
|
||||
loss. Similar to the operation of
|
||||
Basho's
|
||||
[Riak key-value store, Riak KV](http://basho.com/products/riak-kv/),
|
||||
Machi can provide file writes during arbitrary network partitions and
|
||||
later merge all results together safely when the cluster recovers.
|
||||
|
||||
For a much longer answer, please see the
|
||||
[Machi high level design doc](https://github.com/basho/machi/tree/master/doc/high-level-machi.pdf).
|
||||
|
||||
<a name="n1.2">
|
||||
### 1.2. What is a Machi chain?
|
||||
|
||||
A Machi chain is a small number of machines that maintain a common set
|
||||
of replicated files. A typical chain is of length 2 or 3. For
|
||||
critical data that must be available despite several simultaneous
|
||||
server failures, a chain length of 6 or 7 might be used.
|
||||
|
||||
<a name="n1.3">
|
||||
### 1.3. What is a Machi cluster?
|
||||
|
||||
A Machi cluster is a collection of Machi chains that
|
||||
partitions/shards/distributes files (based on file name) across the
|
||||
collection of chains. Machi uses the "random slicing" algorithm (a
|
||||
variation of consistent hashing) to define the mapping of file name to
|
||||
chain name.
|
||||
|
||||
The cluster management service will be fully decentralized
|
||||
and run as a separate software service installed on each Machi
|
||||
cluster. This manager will appear to the local Machi server as simply
|
||||
another Machi file client. The cluster managers will take
|
||||
care of file migration as the cluster grows and shrinks in capacity
|
||||
and in response to day-to-day changes in workload.
|
||||
|
||||
Though the cluster manager has not yet been implemented,
|
||||
its design is fully decentralized and capable of operating despite
|
||||
multiple partial failure of its member chains. We expect this
|
||||
design to scale easily to at least one thousand servers.
|
||||
|
||||
Please see the
|
||||
[Machi source repository's 'doc' directory for more details](https://github.com/basho/machi/tree/master/doc/).
|
||||
|
||||
<a name="n1.4">
|
||||
### 1.4. What is Machi like when operating in "eventually consistent" mode?
|
||||
|
||||
Machi's operating mode dictates how a Machi cluster will react to
|
||||
network partitions. A network partition may be caused by:
|
||||
|
||||
* A network failure
|
||||
* A server failure
|
||||
* An extreme server software "hang" or "pause", e.g. caused by OS
|
||||
scheduling problems such as a failing/stuttering disk device.
|
||||
|
||||
The consistency semantics of file operations while in eventual
|
||||
consistency mode during and after network partitions are:
|
||||
|
||||
* File write operations are permitted by any client on the "same side"
|
||||
of the network partition.
|
||||
* File read operations are successful for any file contents where the
|
||||
client & server are on the "same side" of the network partition.
|
||||
* File read operations will probably fail for any file contents where the
|
||||
client & server are on "different sides" of the network partition.
|
||||
* After the network partition(s) is resolved, files are merged
|
||||
together from "all sides" of the partition(s).
|
||||
* Unique files are copied in their entirety.
|
||||
* Byte ranges within the same file are merged. This is possible
|
||||
due to Machi's restrictions on file naming and file offset
|
||||
assignment. Both file names and file offsets are always chosen
|
||||
by Machi servers according to rules which guarantee safe
|
||||
mergeability. Server-assigned names are a characteristic of a
|
||||
"blob store".
|
||||
|
||||
<a name="n1.5">
|
||||
### 1.5. What is Machi like when operating in "strongly consistent" mode?
|
||||
|
||||
The consistency semantics of file operations while in strongly
|
||||
consistency mode during and after network partitions are:
|
||||
|
||||
* File write operations are permitted by any client on the "same side"
|
||||
of the network partition if and only if a quorum majority of Machi servers
|
||||
are also accessible within that partition.
|
||||
* In other words, file write service is unavailable in any
|
||||
partition where only a minority of Machi servers are accessible.
|
||||
* File read operations are successful for any file contents where the
|
||||
client & server are on the "same side" of the network partition.
|
||||
* After the network partition(s) is resolved, files are repaired from
|
||||
the surviving quorum majority members to out-of-sync minority
|
||||
members.
|
||||
|
||||
Machi's design can provide the illusion of quorum minority write
|
||||
availability if the cluster is configured to operate with "witness
|
||||
servers". (This feaure partially implemented, as of December 2015.)
|
||||
See Section 11 of
|
||||
[Machi chain manager high level design doc](https://github.com/basho/machi/tree/master/doc/high-level-chain-mgr.pdf)
|
||||
for more details.
|
||||
|
||||
<a name="n1.6">
|
||||
### 1.6. What does Machi's API look like?
|
||||
|
||||
The Machi API only contains a handful of API operations. The function
|
||||
arguments shown below (in simplifed form) use Erlang-style type annotations.
|
||||
|
||||
append_chunk(Prefix:binary(), Chunk:binary(), CheckSum:binary()).
|
||||
append_chunk_extra(Prefix:binary(), Chunk:binary(), CheckSum:binary(), ExtraSpace:non_neg_integer()).
|
||||
read_chunk(File:binary(), Offset:non_neg_integer(), Size:non_neg_integer()).
|
||||
|
||||
checksum_list(File:binary()).
|
||||
list_files().
|
||||
|
||||
Machi allows the client to choose the prefix of the file name to
|
||||
append data to, but the Machi server will always choose the final file
|
||||
name and byte offset for each `append_chunk()` operation. This
|
||||
restriction on file naming makes it easy to operate in "eventually
|
||||
consistent" mode: files may be written to any server during network
|
||||
partitions and can be easily merged together after the partition is
|
||||
healed.
|
||||
|
||||
Internally, there is a more complex protocol used by individual
|
||||
cluster members to manage file contents and to repair damaged/missing
|
||||
files. See Figure 3 in
|
||||
[Machi high level design doc](https://github.com/basho/machi/tree/master/doc/high-level-machi.pdf)
|
||||
for more description.
|
||||
|
||||
The definitions of both the "high level" external protocol and "low
|
||||
level" internal protocol are in a
|
||||
[Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview)
|
||||
definition at [./src/machi.proto](./src/machi.proto).
|
||||
|
||||
<a name="n1.7">
|
||||
### 1.7. What licensing terms are used by Machi?
|
||||
|
||||
All Machi source code and documentation is licensed by
|
||||
[Basho Technologies, Inc.](http://www.basho.com/)
|
||||
under the [Apache Public License version 2](https://github.com/basho/machi/tree/master/LICENSE).
|
||||
|
||||
<a name="n1.8">
|
||||
### 1.8. Where can I find the Machi source code and documentation? Can I contribute?
|
||||
|
||||
All Machi source code and documentation can be found at GitHub:
|
||||
[https://github.com/basho/machi](https://github.com/basho/machi).
|
||||
The full URL for this FAQ is [https://github.com/basho/machi/blob/master/FAQ.md](https://github.com/basho/machi/blob/master/FAQ.md).
|
||||
|
||||
There are several "README" files in the source repository. We hope
|
||||
they provide useful guidance for first-time readers.
|
||||
|
||||
If you're interested in contributing code or documentation or
|
||||
ideas for improvement, please see our contributing & collaboration
|
||||
guidelines at
|
||||
[https://github.com/basho/machi/blob/master/CONTRIBUTING.md](https://github.com/basho/machi/blob/master/CONTRIBUTING.md).
|
||||
|
||||
<a name="n1.9">
|
||||
### 1.9. What is Machi's expected release schedule, packaging, and operating system/OS distribution support?
|
||||
|
||||
Basho expects that Machi's first major product release will take place
|
||||
during the 2nd quarter of 2016.
|
||||
|
||||
Basho's official support for operating systems (e.g. Linux, FreeBSD),
|
||||
operating system packaging (e.g. CentOS rpm/yum package management,
|
||||
Ubuntu debian/apt-get package management), and
|
||||
container/virtualization have not yet been chosen. If you wish to
|
||||
provide your opinion, we'd love to hear it. Please
|
||||
[open a support ticket at GitHub](https://github.com/basho/machi/issues/new)
|
||||
and let us know.
|
||||
|
||||
<a name="n2">
|
||||
## 2. Questions about Machi relative to {{something else}}
|
||||
|
||||
<a name="better-than-hadoop">
|
||||
<a name="n2.1">
|
||||
### 2.1. How is Machi better than Hadoop?
|
||||
|
||||
This question is frequently asked by trolls. If this is a troll
|
||||
question, the answer is either, "Nothing is better than Hadoop," or
|
||||
else "Everything is better than Hadoop."
|
||||
|
||||
The real answer is that Machi is not a distributed data processing
|
||||
framework like Hadoop is.
|
||||
See [Hadoop's entry in Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop)
|
||||
and focus on the description of Hadoop's MapReduce and YARN; Machi
|
||||
contains neither.
|
||||
|
||||
<a name="n2.2">
|
||||
### 2.2. How does Machi differ from HadoopFS/HDFS?
|
||||
|
||||
This is a much better question than the
|
||||
[How is Machi better than Hadoop?](#better-than-hadoop)
|
||||
question.
|
||||
|
||||
[HadoopFS's entry in Wikipedia](https://en.wikipedia.org/wiki/Apache_Hadoop#HDFS)
|
||||
|
||||
One way to look at Machi is to consider Machi as a distributed file
|
||||
store. HadoopFS is also a distributed file store. Let's compare and
|
||||
contrast.
|
||||
|
||||
<table>
|
||||
|
||||
<tr>
|
||||
<td> <b>Machi</b>
|
||||
<td> <b>HadoopFS (HDFS)</b>
|
||||
|
||||
<tr>
|
||||
<td> Not POSIX compliant
|
||||
<td> Not POSIX compliant
|
||||
|
||||
<tr>
|
||||
<td> Immutable file store with append-only semantics (simplifying
|
||||
things a little bit).
|
||||
<td> Immutable file store with append-only semantics
|
||||
|
||||
<tr>
|
||||
<td> File data may be read concurrently while file is being actively
|
||||
appended to.
|
||||
<td> File must be closed before a client can read it.
|
||||
|
||||
<tr>
|
||||
<td> No concept (yet) of users or authentication (though the initial
|
||||
supported release will support basic user + password authentication).
|
||||
Machi will probably never natively support directories or ACLs.
|
||||
<td> Has concepts of users, directories, and ACLs.
|
||||
|
||||
<tr>
|
||||
<td> Machi does not allow clients to name their own files or to specify data
|
||||
placement/offset within a file.
|
||||
<td> While not POSIX compliant, HDFS allows a fairly flexible API for
|
||||
managing file names and file writing position within a file (during a
|
||||
file's writable phase).
|
||||
|
||||
<tr>
|
||||
<td> Does not have any file distribution/partitioning/sharding across
|
||||
Machi chains: in a single Machi chain, all files are replicated by
|
||||
all servers in the chain. The "random slicing" technique is used
|
||||
to distribute/partition/shard files across multiple Machi clusters.
|
||||
<td> File distribution/partitioning/sharding is performed
|
||||
automatically by the HDFS "name node".
|
||||
|
||||
<tr>
|
||||
<td> Machi requires no central "name node" for single chain use or
|
||||
for multi-chain cluster use.
|
||||
<td> Requires a single "namenode" server to maintain file system contents
|
||||
and file content mapping. (May be deployed with a "secondary
|
||||
namenode" to reduce unavailability when the primary namenode fails.)
|
||||
|
||||
<tr>
|
||||
<td> Machi uses Chain Replication to manage all file replicas.
|
||||
<td> The HDFS name node uses an ad hoc mechanism for replicating file
|
||||
contents. The HDFS file system metadata (file names, file block(s)
|
||||
locations, ACLs, etc.) is stored by the name node in the local file
|
||||
system and is replicated to any secondary namenode using snapshots.
|
||||
|
||||
<tr>
|
||||
<td> Machi replicates files *N* ways where *N* is the length of the
|
||||
Chain Replication chain. Typically, *N=2*, but this is configurable.
|
||||
<td> HDFS typical replicates file contents *N=3* ways, but this is
|
||||
configurable.
|
||||
|
||||
<tr>
|
||||
<td> All Machi file data is protected by SHA-1 checksums generated by
|
||||
the client prior to writing by Machi servers.
|
||||
<td> Optional file checksum protection may be implemented on the
|
||||
server side.
|
||||
|
||||
</table>
|
||||
|
||||
<a name="n2.3">
|
||||
### 2.3. How does Machi differ from Kafka?
|
||||
|
||||
Machi is rather close to Kafka in spirit, though its implementation is
|
||||
quite different.
|
||||
|
||||
<table>
|
||||
|
||||
<tr>
|
||||
<td> <b>Machi</b>
|
||||
<td> <b>Kafka</b>
|
||||
|
||||
<tr>
|
||||
<td> Append-only, strongly consistent file store only
|
||||
<td> Append-only, strongly consistent log file store + additional
|
||||
services: for example, producer topics & sharding, consumer groups &
|
||||
failover, etc.
|
||||
|
||||
<tr>
|
||||
<td> Not yet code complete nor "battle tested" in large production
|
||||
environments.
|
||||
<td> "Battle tested" in large production environments.
|
||||
|
||||
<tr>
|
||||
<td> All Machi file data is protected by SHA-1 checksums generated by
|
||||
the client prior to writing by Machi servers.
|
||||
<td> Each log entry is protected by a 32 bit CRC checksum.
|
||||
|
||||
</table>
|
||||
|
||||
In theory, it should be "quite straightforward" to remove these parts
|
||||
of Kafka's code base:
|
||||
|
||||
* local file system I/O for all topic/partition/log files
|
||||
* leader/follower file replication, ISR ("In Sync Replica") state
|
||||
management, and related log file replication logic
|
||||
|
||||
... and replace those parts with Machi client API calls. Those parts
|
||||
of Kafka are what Machi has been designed to do from the very
|
||||
beginning.
|
||||
|
||||
See also:
|
||||
<a href="#corfu-and-tango">How does Machi differ from CORFU and Tango?</a>
|
||||
|
||||
<a name="n2.4">
|
||||
### 2.4. How does Machi differ from Bookkeeper?
|
||||
|
||||
Sorry, we haven't studied Bookkeeper very deeply or used Bookkeeper
|
||||
for any non-trivial project.
|
||||
|
||||
One notable limitation of the Bookkeeper API is that a ledger cannot
|
||||
be read by other clients until it has been closed. Any byte in a
|
||||
Machi file that has been written successfully may
|
||||
be read immedately by any other Machi client.
|
||||
|
||||
The name "Machi" does not have three consecutive pairs of repeating
|
||||
letters. The name "Bookkeeper" does.
|
||||
|
||||
<a name="corfu-and-tango">
|
||||
<a name="n2.5">
|
||||
### 2.5. How does Machi differ from CORFU and Tango?
|
||||
|
||||
Machi's design borrows very heavily from CORFU. We acknowledge a deep
|
||||
debt to the original Microsoft Research papers that describe CORFU's
|
||||
original design and implementation.
|
||||
|
||||
<table>
|
||||
|
||||
<tr>
|
||||
<td> <b>Machi</b>
|
||||
<td> <b>CORFU</b>
|
||||
|
||||
<tr>
|
||||
<td> Writes & reads may be on byte boundaries
|
||||
<td> Wries & reads must be on page boundaries, e.g. 4 or 8 KBytes, to
|
||||
align with server storage based on flash NVRAM/solid state disk (SSD).
|
||||
|
||||
<tr>
|
||||
<td> Provides multiple "logs", where each log has a name and is
|
||||
appended to & read from like a file. A read operation requires a 3-tuple:
|
||||
file name, starting byte offset, number of bytes.
|
||||
<td> Provides a single "log". A read operation requires only a
|
||||
1-tuple: the log page number. (A protocol option exists to
|
||||
request multiple pages in a single read query?)
|
||||
|
||||
<tr>
|
||||
<td> Offers service in either strongly consistent mode or eventually
|
||||
consistent mode.
|
||||
<td> Offers service in strongly consistent mode.
|
||||
|
||||
<tr>
|
||||
<td> May be deployed on solid state disk (SSD) or Winchester hard disks.
|
||||
<td> Designed for use with solid state disk (SSD) but can also be used
|
||||
with Winchester hard disks (with a performance penalty if used as
|
||||
suggested by use cases described by the CORFU papers).
|
||||
|
||||
<tr>
|
||||
<td> All Machi file data is protected by SHA-1 checksums generated by
|
||||
the client prior to writing by Machi servers.
|
||||
<td> Depending on server & flash device capabilities, each data page
|
||||
may be protected by a checksum (calculated independently by each
|
||||
server rather than the client).
|
||||
|
||||
</table>
|
||||
|
||||
See also: the "Recommended reading & related work" and "References"
|
||||
sections of the
|
||||
[Machi high level design doc](https://github.com/basho/machi/tree/master/doc/high-level-machi.pdf)
|
||||
for pointers to the MSR papers related to CORFU.
|
||||
|
||||
Machi does not implement Tango directly. (Not yet, at least.)
|
||||
However, there is a prototype implementation included in the Machi
|
||||
source tree. See
|
||||
[the prototype/tango source code directory](https://github.com/basho/machi/tree/master/prototype/tango)
|
||||
for details.
|
||||
|
||||
Also, it's worth adding that the original MSR code behind the research
|
||||
papers is now available at GitHub:
|
||||
[https://github.com/CorfuDB/CorfuDB](https://github.com/CorfuDB/CorfuDB).
|
||||
|
||||
<a name="n3">
|
||||
## 3. Machi's specifics
|
||||
|
||||
<a name="n3.1">
|
||||
### 3.1. What technique is used to replicate Machi's files? Can other techniques be used?
|
||||
|
||||
Machi uses Chain Replication to replicate all file data. Each byte of
|
||||
a file is stored using a "write-once register", which is a mechanism
|
||||
to enforce immutability after the byte has been written exactly once.
|
||||
|
||||
In order to ensure availability in the event of *F* failures, Chain
|
||||
Replication requires a minimum of *F + 1* servers to be configured.
|
||||
|
||||
Alternative mechanisms could be used to manage file replicas, such as
|
||||
Paxos or Raft. Both Paxos and Raft have some requirements that are
|
||||
difficult to adapt to Machi's design goals:
|
||||
|
||||
* Both protocols use quorum majority consensus, which requires a
|
||||
minimum of *2F + 1* working servers to tolerate *F* failures. For
|
||||
example, to tolerate 2 server failures, quorum majority protocols
|
||||
require a minimum of 5 servers. To tolerate the same number of
|
||||
failures, Chain Replication requires a minimum of only 3 servers.
|
||||
* Machi's use of "humming consensus" to manage internal server
|
||||
metadata state would also (probably) require conversion to Paxos or
|
||||
Raft. (Or "outsourced" to a service such as ZooKeeper.)
|
||||
|
||||
<a name="n3.2">
|
||||
### 3.2. Does Machi have a reliance on a coordination service such as ZooKeeper or etcd?
|
||||
|
||||
No. Machi maintains critical internal cluster information in an
|
||||
internal, immutable data service called the "projection store". The
|
||||
contents of the projection store are maintained by a new technique
|
||||
called "humming consensus".
|
||||
|
||||
Humming consensus is described in the
|
||||
[Machi chain manager high level design doc](https://github.com/basho/machi/tree/master/doc/high-level-chain-mgr.pdf).
|
||||
|
||||
<a name="n3.3">
|
||||
### 3.3. Are there any presentations available about Humming Consensus
|
||||
|
||||
Scott recently (November 2015) gave a presentation at the
|
||||
[RICON 2015 conference](http://ricon.io) about one of the techniques
|
||||
used by Machi; "Managing Chain Replication Metadata with
|
||||
Humming Consensus" is available online now.
|
||||
* [slides (PDF format)](http://ricon.io/speakers/slides/Scott_Fritchie_Ricon_2015.pdf)
|
||||
* [video](https://www.youtube.com/watch?v=yR5kHL1bu1Q)
|
||||
|
||||
<a name="n3.4">
|
||||
### 3.4. Is it true that there's an allegory written to describe Humming Consensus?
|
||||
|
||||
Yes. In homage to Leslie Lamport's original paper about the Paxos
|
||||
protocol, "The Part-time Parliamant", there is an allegorical story
|
||||
that describes humming consensus as method to coordinate
|
||||
many composers to write a single piece of music.
|
||||
The full story, full of wonder and mystery, is called
|
||||
["On “Humming Consensus”, an allegory"](http://www.snookles.com/slf-blog/2015/03/01/on-humming-consensus-an-allegory/).
|
||||
There is also a
|
||||
[short followup blog posting](http://www.snookles.com/slf-blog/2015/03/20/on-humming-consensus-an-allegory-part-2/).
|
||||
|
||||
<a name="n3.5">
|
||||
### 3.5. How is Machi tested?
|
||||
|
||||
While not formally proven yet, Machi's implementation of Chain
|
||||
Replication and of humming consensus have been extensively tested with
|
||||
several techniques:
|
||||
|
||||
* We use an executable model based on the QuickCheck framework for
|
||||
property based testing.
|
||||
* In addition to QuickCheck alone, we use the PULSE extension to
|
||||
QuickCheck is used to test the implementation
|
||||
under extremely skewed & unfair scheduling/timing conditions.
|
||||
|
||||
The model includes simulation of asymmetric network partitions. For
|
||||
example, actor A can send messages to actor B, but B cannot send
|
||||
messages to A. If such a partition happens somewhere in a traditional
|
||||
network stack (e.g. a faulty Ethernet cable), any TCP connection
|
||||
between A & B will quickly interrupt communication in _both_
|
||||
directions. In the Machi network partition simulator, network
|
||||
partitions can be truly one-way only.
|
||||
|
||||
After randomly generating a series of network partitions (which may
|
||||
change several times during any single test case) and a random series
|
||||
of cluster operations, an event trace of all cluster activity is used
|
||||
to verify that no safety-critical rules have been violated.
|
||||
|
||||
All test code is available in the [./test](./test) subdirectory.
|
||||
Modules that use QuickCheck will use a file suffix of `_eqc`, for
|
||||
example, [./test/machi_ap_repair_eqc.erl](./test/machi_ap_repair_eqc.erl).
|
||||
|
||||
<a name="n3.6">
|
||||
### 3.6. Does Machi require shared disk storage? e.g. iSCSI, NBD (Network Block Device), Fibre Channel disks
|
||||
|
||||
No, Machi's design assumes that each Machi server is a fully
|
||||
independent hardware and assumes only standard local disks (Winchester
|
||||
and/or SSD style) with local-only interfaces (e.g. SATA, SCSI, PCI) in
|
||||
each machine.
|
||||
|
||||
<a name="n3.7">
|
||||
### 3.7. Does Machi require or assume that servers with large numbers of disks must use RAID-0/1/5/6/10/50/60 to create a single block device?
|
||||
|
||||
No. When used with servers with multiple disks, the intent is to
|
||||
deploy multiple Machi servers per machine: one Machi server per disk.
|
||||
|
||||
* Pro: disk bandwidth and disk storage capacity can be managed at the
|
||||
level of an individual disk.
|
||||
* Pro: failure of an individual disk does not risk data loss on other
|
||||
disks.
|
||||
* Con (or pro, depending on the circumstances): in this configuration,
|
||||
Machi would require additional network bandwidth to repair data on a lost
|
||||
drive instead of intra-machine disk & bus & memory bandwidth that
|
||||
would be required for RAID volume repair
|
||||
* Con: replica placement policy, such as "rack awareness", becomes a
|
||||
larger problem that must be automated. For example, a problem of
|
||||
placement relative to 12 servers is smaller than a placement problem
|
||||
of managing 264 seprate disks (if each of 12 servers has 22 disks).
|
||||
|
||||
<a name="n3.8">
|
||||
### 3.8. What language(s) is Machi written in?
|
||||
|
||||
So far, Machi is written in Erlang, mostly. Machi uses at least one
|
||||
library, [ELevelDB](https://github.com/basho/eleveldb), that is
|
||||
implemented both in C++ and in Erlang, using Erlang NIFs (Native
|
||||
Interface Functions) to allow Erlang code to call C++ functions.
|
||||
|
||||
In the event that we encounter a performance problem that cannot be
|
||||
solved within the Erlang/OTP runtime environment, all of Machi's
|
||||
performance-critical components are small enough to be re-implemented
|
||||
in C, Java, or other "gotta go fast fast FAST!!" programming
|
||||
language. We expect that the Chain Replication manager and other
|
||||
critical "control plane" software will remain in Erlang.
|
||||
|
||||
<a name="n3.9">
|
||||
### 3.9. Can Machi run on Windows? Can Machi run on 32-bit platforms?
|
||||
|
||||
The ELevelDB NIF does not compile or run correctly on Erlang/OTP
|
||||
Windows platforms, nor does it compile correctly on 32-bit platforms.
|
||||
Machi should support all 64-bit UNIX-like platforms that are supported
|
||||
by Erlang/OTP and ELevelDB.
|
||||
|
||||
<a name="n3.10">
|
||||
### 3.10. Does Machi use the Erlang/OTP network distribution system (aka "disterl")?
|
||||
|
||||
No, Machi doesn't use Erlang/OTP's built-in distributed message
|
||||
passing system. The code would be *much* simpler if we did use
|
||||
"disterl". However, due to (premature?) worries about performance, we
|
||||
wanted to have the option of re-writing some Machi components in C or
|
||||
Java or Go or OCaml or COBOL or in-kernel assembly hexadecimal
|
||||
bit-twiddling magicSPEED ... without also having to find a replacement
|
||||
for disterl. (Or without having to re-invent disterl's features in
|
||||
another language.)
|
||||
|
||||
All wire protocols used by Machi are defined & implemented using
|
||||
[Protocol Buffers](https://developers.google.com/protocol-buffers/docs/overview).
|
||||
The definition file can be found at [./src/machi.proto](./src/machi.proto).
|
||||
|
||||
<a name="n3.11">
|
||||
### 3.11. Can I use HTTP to write/read stuff into/from Machi?
|
||||
|
||||
Short answer: No, not yet.
|
||||
|
||||
Longer answer: No, but it was possible as a hack, many months ago, see
|
||||
[primitive/hack'y HTTP interface that is described in this source code commit log](https://github.com/basho/machi/commit/6cebf397232cba8e63c5c9a0a8c02ba391b20fef).
|
||||
Please note that commit `6cebf397232cba8e63c5c9a0a8c02ba391b20fef` is
|
||||
required to try using this feature: the code has since bit-rotted and
|
||||
will not work on today's `master` branch.
|
||||
|
||||
In the long term, we'll probably want the option of an HTTP interface
|
||||
that is as well designed and REST'ful as possible. It's on the
|
||||
internal Basho roadmap. If you'd like to work on a real, not-kludgy
|
||||
HTTP interface to Machi,
|
||||
[please contact us!](https://github.com/basho/machi/blob/master/CONTRIBUTING.md)
|
||||
|
||||
297
INSTALLATION.md
297
INSTALLATION.md
|
|
@ -1,297 +0,0 @@
|
|||
|
||||
# Installation instructions for Machi
|
||||
|
||||
Machi is still a young enough project that there is no "installation".
|
||||
All development is still done using the Erlang/OTP interactive shell
|
||||
for experimentation, using `make` to compile, and the shell's
|
||||
`l(ModuleName).` command to reload any recompiled modules.
|
||||
|
||||
In the coming months (mid-2015), there are plans to create OS packages
|
||||
for common operating systems and OS distributions, such as FreeBSD and
|
||||
Linux. If building RPM, DEB, PKG, or other OS package managers is
|
||||
your specialty, we could use your help to speed up the process! <b>:-)</b>
|
||||
|
||||
## Development toolchain dependencies
|
||||
|
||||
Machi's dependencies on the developer's toolchain are quite small.
|
||||
|
||||
* Erlang/OTP version 17.0 or later, 32-bit or 64-bit
|
||||
* The `make` utility.
|
||||
* The GNU version of make is not required.
|
||||
* Machi is bundled with a `rebar` package and should be usable on
|
||||
any Erlang/OTP 17.x platform.
|
||||
|
||||
Machi does not use any Erlang NIF or port drivers.
|
||||
|
||||
## Development OS
|
||||
|
||||
At this time, Machi is 100% Erlang. Although we have not tested it,
|
||||
there should be no good reason why Machi cannot run on Erlang/OTP on
|
||||
Windows platforms. Machi has been developed on OS X and FreeBSD and
|
||||
is expected to work on any UNIX-ish platform supported by Erlang/OTP.
|
||||
|
||||
## Compiling the Machi source
|
||||
|
||||
First, clone the Machi source code, then compile it. You will
|
||||
need Erlang/OTP version 17.x to compile.
|
||||
|
||||
cd /some/nice/dev/place
|
||||
git clone https://github.com/basho/machi.git
|
||||
cd machi
|
||||
make
|
||||
make test
|
||||
|
||||
The unit test suite is based on the EUnit framework (bundled with
|
||||
Erlang/OTP 17). The `make test` suite runs on my MacBook in 10
|
||||
seconds or less.
|
||||
|
||||
## Setting up a Machi cluster
|
||||
|
||||
As noted above, everything is done manually at the moment. Here is a
|
||||
rough sketch of day-to-day development workflow.
|
||||
|
||||
### 1. Run the server
|
||||
|
||||
cd /some/nice/dev/place/machi
|
||||
make
|
||||
erl -pz ebin deps/*/ebin +A 253 +K true
|
||||
|
||||
This will start an Erlang shell, plus a few extras.
|
||||
|
||||
* Tell the OTP code loader where to find dependent BEAM files.
|
||||
* Set a large pool (253) of file I/O worker threads
|
||||
* Use a more efficient kernel polling mechanism for network sockets.
|
||||
* If your Erlang/OTP package does not support `+K true`, do not
|
||||
worry. It is an optional flag.
|
||||
|
||||
The following commands will start three Machi FLU server processes and
|
||||
then tell them to form a single chain. Internally, each FLU will have
|
||||
Erlang registered processes with the names `a`, `b`, and `c`, and
|
||||
listen on TCP ports 4444, 4445, and 4446, respectively. Each will use
|
||||
a data directory located in the current directory, e.g. `./data.a`.
|
||||
|
||||
Cut-and-paste the following commands into the CLI at the prompt:
|
||||
|
||||
application:ensure_all_started(machi).
|
||||
machi_flu_psup:start_flu_package(a, 4444, "./data.a", []).
|
||||
machi_flu_psup:start_flu_package(b, 4445, "./data.b", []).
|
||||
D = orddict:from_list([{a,{p_srvr,a,machi_flu1_client,"localhost",4444,[]}},{b,{p_srvr,b,machi_flu1_client,"localhost",4445,[]}}]).
|
||||
machi_chain_manager1:set_chain_members(a_chmgr, D).
|
||||
machi_chain_manager1:set_chain_members(b_chmgr, D).
|
||||
|
||||
If you change the TCP ports of any of the processes, you must make the
|
||||
same change both in the `machi_flu_psup:start_flu_package()` arguments
|
||||
and also in the `D` dictionary.
|
||||
|
||||
The Erlang processes that will be started are arranged in the
|
||||
following hierarchy. See the
|
||||
[machi_flu_psup.erl](http://basho.github.io/machi/edoc/machi_flu_psup.html)
|
||||
EDoc documentation for a description of each of these processes.
|
||||
|
||||
|
||||
|
||||
### 2. Check the status of the server processes.
|
||||
|
||||
Each Machi FLU is an independent file server. All replication between
|
||||
Machi servers is currently implemented by code on the *client* side.
|
||||
(This will change a bit later in 2015.)
|
||||
|
||||
Use the `read_latest_projection` command on the server CLI, e.g.:
|
||||
|
||||
rr("include/machi_projection.hrl").
|
||||
machi_projection_store:read_latest_projection(a_pstore, private).
|
||||
|
||||
... to query the projection store of the local FLU named `a`.
|
||||
|
||||
If you haven't looked at the server-side description of the various
|
||||
Machi server-side processes, please take a couple minutes to read
|
||||
[machi_flu_psup.erl](http://basho.github.io/machi/edoc/machi_flu_psup.html).
|
||||
|
||||
### 3. Use the machi_cr_client.erl client
|
||||
|
||||
For development work, I run the client & server on the same Erlang
|
||||
VM. It's just easier that way ... but the Machi client & server use
|
||||
TCP to communicate with each other.
|
||||
|
||||
If you are using a separate machine for the client, then compile the
|
||||
Machi source on the client machine. Then run:
|
||||
|
||||
cd /some/nice/dev/place/machi
|
||||
make
|
||||
erl -pz ebin deps/*/ebin
|
||||
|
||||
(You can add `+K true` if you wish ... but for light development work,
|
||||
it doesn't make a big difference.)
|
||||
|
||||
At the CLI, define the dictionary that describes the host & TCP port
|
||||
location for each of the Machi servers. (If you changed the host
|
||||
and/or TCP port values when starting the servers, then place make the
|
||||
same changes here.
|
||||
|
||||
D = orddict:from_list([{a,{p_srvr,a,machi_flu1_client,"localhost",4444,[]}},{b,{p_srvr,b,machi_flu1_client,"localhost",4445,[]}}]).
|
||||
|
||||
Then start a `machi_cr_client` client process.
|
||||
|
||||
{ok, C1} = machi_cr_client:start_link([P || {_,P} <- orddict:to_list(D)]).
|
||||
|
||||
Please keep in mind that this process is **linked** to your CLI
|
||||
process. If you run a CLI command the throws an exception/exits, then
|
||||
this `C1` process will also die! You can start a new one, using a
|
||||
different name, e.g. `C2`. Or you can start a new one by first
|
||||
"forgetting" the CLI's binding for `C1`.
|
||||
|
||||
f(C1).
|
||||
{ok, C1} = machi_cr_client:start_link([P || {_,P} <- orddict:to_list(D)]).
|
||||
|
||||
Now, append a small chunk of data to a file with the prefix
|
||||
`<<"pre">>`.
|
||||
|
||||
12> {ok, C1} = machi_cr_client:start_link([P || {_,P} <- orddict:to_list(D)]).
|
||||
{ok,<0.112.0>}
|
||||
|
||||
13> machi_cr_client:append_chunk(C1, <<"pre">>, <<"Hello, world">>).
|
||||
{ok,{1024,12,<<"pre.G6C116EA.3">>}}
|
||||
|
||||
|
||||
This chunk was written successfully to a file called
|
||||
`<<"pre.5BBL16EA.1">>` at byte offset 1024. Let's fetch it now. And
|
||||
let's see what happens in a couple of error conditions: fetching
|
||||
bytes that "straddle" the end of file, bytes that are after the known
|
||||
end of file, and bytes from a file that has never been written.
|
||||
|
||||
26> machi_cr_client:read_chunk(C1, <<"pre.G6C116EA.3">>, 1024, 12).
|
||||
{ok,<<"Hello, world">>}
|
||||
|
||||
27> machi_cr_client:read_chunk(C1, <<"pre.G6C116EA.3">>, 1024, 777).
|
||||
{error,partial_read}
|
||||
|
||||
28> machi_cr_client:read_chunk(C1, <<"pre.G6C116EA.3">>, 889323, 12).
|
||||
{error,not_written}
|
||||
|
||||
29> machi_cr_client:read_chunk(C1, <<"no-such-file">>, 1024, 12).
|
||||
{error,not_written}
|
||||
|
||||
### 4. Use the `machi_proxy_flu1_client.erl` client
|
||||
|
||||
The `machi_proxy_flu1_client` module implements a simpler client that
|
||||
only uses a single Machi FLU file server. This client is **not**
|
||||
aware of chain replication in any way.
|
||||
|
||||
Let's use this client to verify that the `<<"Hello, world!">> data
|
||||
that we wrote in step #3 was truly written to both FLU servers by the
|
||||
`machi_cr_client` library. We start proxy processes for each of the
|
||||
FLUs, then we'll query each ... but first we also need to ask (at
|
||||
least one of) the servers for the current Machi cluster's Epoch ID.
|
||||
|
||||
{ok, Pa} = machi_proxy_flu1_client:start_link(orddict:fetch(a, D)).
|
||||
{ok, Pb} = machi_proxy_flu1_client:start_link(orddict:fetch(b, D)).
|
||||
{ok, EpochID0} = machi_proxy_flu1_client:get_epoch_id(Pa).
|
||||
machi_proxy_flu1_client:read_chunk(Pa, EpochID0, <<"pre.G6C116EA.3">>, 1024, 12).
|
||||
machi_proxy_flu1_client:read_chunk(Pb, EpochID0, <<"pre.G6C116EA.3">>, 1024, 12).
|
||||
|
||||
### 5. Checking how Chain Replication "read repair" works
|
||||
|
||||
Now, let's cause some trouble: we will write some data only to the
|
||||
head of the chain. By default, all read operations go to the tail of
|
||||
the chain. But, if a value is not written at the tail, then "read
|
||||
repair" ought to verify:
|
||||
|
||||
* Perhaps the value truly is not written at any server in the chain.
|
||||
* Perhaps the value was partially written, i.e. by a buggy or
|
||||
crashed-in-the-middle-of-the-writing-procedure client.
|
||||
|
||||
So, first, let's double-check that the chain is in the order that we
|
||||
expect it to be.
|
||||
|
||||
rr("include/machi_projection.hrl"). % In case you didn't do this earlier.
|
||||
machi_proxy_flu1_client:read_latest_projection(Pa, private).
|
||||
|
||||
The part of the `#projection_v1` record that we're interested in is
|
||||
the `upi`. This is the list of servers that preserve the Update
|
||||
Propagation Invariant property of the Chain Replication algorithm.
|
||||
The output should look something like:
|
||||
|
||||
{ok,#projection_v1{
|
||||
epoch_number = 1119,
|
||||
[...]
|
||||
author_server = b,
|
||||
all_members = [a,b],
|
||||
creation_time = {1432,189599,85392},
|
||||
mode = ap_mode,
|
||||
upi = [a,b],
|
||||
repairing = [],down = [],
|
||||
[...]
|
||||
}
|
||||
|
||||
So, we see `upi=[a,b]`, which means that FLU `a` is the head of the
|
||||
chain and that `b` is the tail.
|
||||
|
||||
Let's append to `a` using the `machi_proxy_flu1_client` to the head
|
||||
and then read from both the head and tail. (If your chain order is
|
||||
different, then please exchange `Pa` and `Pb` in all of the commands
|
||||
below.)
|
||||
|
||||
16> {ok, {Off1,Size1,File1}} = machi_proxy_flu1_client:append_chunk(Pa, EpochID0, <<"foo">>, <<"Hi, again">>).
|
||||
{ok,{1024,9,<<"foo.K63D16M4.1">>}}
|
||||
|
||||
17> machi_proxy_flu1_client:read_chunk(Pa, EpochID0, File1, Off1, Size1). {ok,<<"Hi, again">>}
|
||||
|
||||
18> machi_proxy_flu1_client:read_chunk(Pb, EpochID0, File1, Off1, Size1).
|
||||
{error,not_written}
|
||||
|
||||
That is correct! Now, let's read the same file & offset using the
|
||||
client that understands chain replication. Then we will try reading
|
||||
directly from FLU `b` again ... we should see something different.
|
||||
|
||||
19> {ok, C2} = machi_cr_client:start_link([P || {_,P} <- orddict:to_list(D)]).
|
||||
{ok,<0.113.0>}
|
||||
|
||||
20> machi_cr_client:read_chunk(C2, File1, Off1, Size1).
|
||||
{ok,<<"Hi, again">>}
|
||||
|
||||
21> machi_proxy_flu1_client:read_chunk(Pb, EpochID0, File1, Off1, Size1).
|
||||
{ok,<<"Hi, again">>}
|
||||
|
||||
That is correct! The command at prompt #20 automatically performed
|
||||
"read repair" on FLU `b`.
|
||||
|
||||
### 6. Exploring what happens when a server is stopped, data written, and server restarted
|
||||
|
||||
Cut-and-paste the following into your CLI. We assume that your CLI
|
||||
still remembers the value of the `D` dictionary from the previous
|
||||
steps above. We will stop FLU `a`, write one thousand small
|
||||
chunks, then restart FLU `a`, then see what happens.
|
||||
|
||||
{ok, C3} = machi_cr_client:start_link([P || {_,P} <- orddict:to_list(D)]).
|
||||
machi_flu_psup:stop_flu_package(a).
|
||||
[machi_cr_client:append_chunk(C3, <<"foo">>, <<"Lots of stuff">>) || _ <- lists:seq(1,1000)].
|
||||
machi_flu_psup:start_flu_package(a, 4444, "./data.a", []).
|
||||
|
||||
About 10 seconds after we restarting `a` with the
|
||||
`machi_flu_psup:start_flu_package()` function, this appears on the
|
||||
console:
|
||||
|
||||
=INFO REPORT==== 21-May-2015::15:53:46 ===
|
||||
Repair start: tail b of [b] -> [a], ap_mode ID {b,{1432,191226,707262}}
|
||||
MissingFileSummary [{<<"foo.CYLJ16ZT.1">>,{14024,[a]}}]
|
||||
Make repair directives: . done
|
||||
Out-of-sync data for FLU a: 0.1 MBytes
|
||||
Out-of-sync data for FLU b: 0.0 MBytes
|
||||
Execute repair directives: .......... done
|
||||
|
||||
=INFO REPORT==== 21-May-2015::15:53:47 ===
|
||||
Repair success: tail b of [b] finished ap_mode repair ID {b,{1432,191226,707262}}: ok
|
||||
Stats [{t_in_files,0},{t_in_chunks,1000},{t_in_bytes,13000},{t_out_files,0},{t_out_chunks,1000},{t_out_bytes,13000},{t_bad_chunks,0},{t_elapsed_seconds,0.647}]
|
||||
|
||||
The data repair process, executed by `b`'s chain manager, found 1000
|
||||
chunks that were out of sync and copied them to `a` successfully.
|
||||
|
||||
### 7. Exploring the rest of the client APIs
|
||||
|
||||
Please see the EDoc documentation for the client APIs. Feel free to
|
||||
explore!
|
||||
|
||||
* [Erlang type definitions for the client APIs](http://basho.github.io/machi/edoc/machi_flu1_client.html)
|
||||
* [EDoc for machi_cr_client.erl](http://basho.github.io/machi/edoc/machi_cr_client.html)
|
||||
* [EDoc for machi_proxy_flu1_client.erl](http://basho.github.io/machi/edoc/machi_proxy_flu1_client.html)
|
||||
* [Top level EDoc collection](http://basho.github.io/machi/edoc/)
|
||||
178
LICENSE
178
LICENSE
|
|
@ -1,178 +0,0 @@
|
|||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
94
Makefile
94
Makefile
|
|
@ -1,94 +0,0 @@
|
|||
REPO ?= machi
|
||||
PKG_REVISION ?= $(shell git describe --tags)
|
||||
PKG_BUILD = 1
|
||||
BASE_DIR = $(shell pwd)
|
||||
ERLANG_BIN = $(shell dirname $(shell which erl))
|
||||
REBAR := $(shell which rebar)
|
||||
ifeq ($(REBAR),)
|
||||
REBAR = $(BASE_DIR)/rebar
|
||||
endif
|
||||
OVERLAY_VARS ?=
|
||||
EUNIT_OPTS = -v
|
||||
|
||||
.PHONY: rel stagedevrel deps package pkgclean edoc
|
||||
|
||||
all: deps compile
|
||||
|
||||
compile:
|
||||
$(REBAR) compile
|
||||
|
||||
## Make reltool happy by creating a fake entry in the deps dir for
|
||||
## machi, because reltool really wants to have a path with
|
||||
## "machi/ebin" at the end, but we also don't want infinite recursion
|
||||
## if we just symlink "deps/machi" -> ".."
|
||||
|
||||
generate:
|
||||
rm -rf deps/machi
|
||||
mkdir deps/machi
|
||||
ln -s ../../ebin deps/machi
|
||||
ln -s ../../src deps/machi
|
||||
$(REBAR) generate $(OVERLAY_VARS) 2>&1 | grep -v 'command does not apply to directory'
|
||||
|
||||
deps:
|
||||
$(REBAR) get-deps
|
||||
|
||||
clean:
|
||||
$(REBAR) -r clean
|
||||
|
||||
edoc: edoc-clean
|
||||
$(REBAR) skip_deps=true doc
|
||||
|
||||
edoc-clean:
|
||||
rm -f edoc/*.png edoc/*.html edoc/*.css edoc/edoc-info
|
||||
|
||||
pulse: compile
|
||||
@echo Sorry, PULSE test needs maintenance. -SLF
|
||||
#env USE_PULSE=1 $(REBAR) skip_deps=true clean compile
|
||||
#env USE_PULSE=1 $(REBAR) skip_deps=true -D PULSE eunit -v
|
||||
|
||||
##
|
||||
## Release targets
|
||||
##
|
||||
rel: deps compile generate
|
||||
|
||||
relclean:
|
||||
rm -rf rel/$(REPO)
|
||||
|
||||
stage : rel
|
||||
$(foreach dep,$(wildcard deps/*), rm -rf rel/$(REPO)/lib/$(shell basename $(dep))* && ln -sf $(abspath $(dep)) rel/$(REPO)/lib;)
|
||||
|
||||
##
|
||||
## Developer targets
|
||||
##
|
||||
## devN - Make a dev build for node N
|
||||
## stagedevN - Make a stage dev build for node N (symlink libraries)
|
||||
## devrel - Make a dev build for 1..$DEVNODES
|
||||
## stagedevrel Make a stagedev build for 1..$DEVNODES
|
||||
##
|
||||
## Example, make a 68 node devrel cluster
|
||||
## make stagedevrel DEVNODES=68
|
||||
|
||||
.PHONY : stagedevrel devrel
|
||||
DEVNODES ?= 3
|
||||
|
||||
# 'seq' is not available on all *BSD, so using an alternate in awk
|
||||
SEQ = $(shell awk 'BEGIN { for (i = 1; i < '$(DEVNODES)'; i++) printf("%i ", i); print i ;exit(0);}')
|
||||
|
||||
$(eval stagedevrel : $(foreach n,$(SEQ),stagedev$(n)))
|
||||
$(eval devrel : $(foreach n,$(SEQ),dev$(n)))
|
||||
|
||||
dev% : all
|
||||
mkdir -p dev
|
||||
rel/gen_dev $@ rel/vars/dev_vars.config.src rel/vars/$@_vars.config
|
||||
(cd rel && ../rebar generate target_dir=../dev/$@ overlay_vars=vars/$@_vars.config)
|
||||
|
||||
stagedev% : dev%
|
||||
$(foreach dep,$(wildcard deps/*), rm -rf dev/$^/lib/$(shell basename $(dep))* && ln -sf $(abspath $(dep)) dev/$^/lib;)
|
||||
|
||||
devclean: clean
|
||||
rm -rf dev
|
||||
|
||||
DIALYZER_APPS = kernel stdlib sasl erts ssl compiler eunit crypto public_key syntax_tools
|
||||
PLT = $(HOME)/.machi_dialyzer_plt
|
||||
|
||||
include tools.mk
|
||||
6
NOTICE
6
NOTICE
|
|
@ -1,6 +0,0 @@
|
|||
Machi
|
||||
Copyright 2007-2015 Basho Technologies
|
||||
|
||||
This product contains code developed at Basho Technologies.
|
||||
(http://www.basho.com/)
|
||||
|
||||
174
README.md
174
README.md
|
|
@ -1,174 +0,0 @@
|
|||
# Machi: a distributed, decentralized blob/large file store
|
||||
|
||||
[Travis-CI](http://travis-ci.org/basho/machi) :: 
|
||||
|
||||
Outline
|
||||
|
||||
1. [Why another blob/file store?](#sec1)
|
||||
2. [Where to learn more about Machi](#sec2)
|
||||
3. [Development status summary](#sec3)
|
||||
4. [Contributing to Machi's development](#sec4)
|
||||
|
||||
<a name="sec1">
|
||||
## 1. Why another blob/file store?
|
||||
|
||||
Our goal is a robust & reliable, distributed, highly available, large
|
||||
file and blob store. Such stores already exist, both in the open source world
|
||||
and in the commercial world. Why reinvent the wheel? We believe
|
||||
there are three reasons, ordered by decreasing rarity.
|
||||
|
||||
1. We want end-to-end checksums for all file data, from the initial
|
||||
file writer to every file reader, anywhere, all the time.
|
||||
2. We need flexibility to trade consistency for availability:
|
||||
e.g. weak consistency in exchange for being available in cases
|
||||
of partial system failure.
|
||||
3. We want to manage file replicas in a way that's provably correct
|
||||
and also easy to test.
|
||||
|
||||
Criteria #3 is difficult to find in the open source world but perhaps
|
||||
not impossible.
|
||||
|
||||
If we have app use cases where availability is more important than
|
||||
consistency, then systems that meet criteria #2 are also rare.
|
||||
Most file stores provide only strong consistency and therefore
|
||||
have unavoidable, unavailable behavior when parts of the system
|
||||
fail.
|
||||
What if we want a file store that is always available to write new
|
||||
file data and attempts best-effort file reads?
|
||||
|
||||
If we really do care about data loss and/or data corruption, then we
|
||||
really want both #3 and #1. Unfortunately, systems that meet
|
||||
criteria #1 are _very rare_. (Nonexistant?)
|
||||
Why? This is 2015. We have decades of research that shows
|
||||
that computer hardware can (and
|
||||
indeed does) corrupt data at nearly every level of the modern
|
||||
client/server application stack. Systems with end-to-end data
|
||||
corruption detection should be ubiquitous today. Alas, they are not.
|
||||
|
||||
Machi is an effort to change the deplorable state of the world, one
|
||||
Erlang function at a time.
|
||||
|
||||
<a name="sec2">
|
||||
## 2. Where to learn more about Machi
|
||||
|
||||
The two major design documents for Machi are now mostly stable.
|
||||
Please see the [doc](./doc) directory's [README](./doc) for details.
|
||||
|
||||
We also have a
|
||||
[Frequently Asked Questions (FAQ) list](./FAQ.md).
|
||||
|
||||
Scott recently (November 2015) gave a presentation at the
|
||||
[RICON 2015 conference](http://ricon.io) about one of the techniques
|
||||
used by Machi; "Managing Chain Replication Metadata with
|
||||
Humming Consensus" is available online now.
|
||||
* [slides (PDF format)](http://ricon.io/speakers/slides/Scott_Fritchie_Ricon_2015.pdf)
|
||||
* [video](https://www.youtube.com/watch?v=yR5kHL1bu1Q)
|
||||
|
||||
See later in this document for how to run the Humming Consensus demos,
|
||||
including the network partition simulator.
|
||||
|
||||
<a name="sec3">
|
||||
## 3. Development status summary
|
||||
|
||||
Mid-March 2016: The Machi development team has been downsized in
|
||||
recent months, and the pace of development has slowed. Here is a
|
||||
summary of the status of Machi's major components.
|
||||
|
||||
* Humming Consensus and the chain manager
|
||||
* No new safety bugs have been found by model-checking tests.
|
||||
* A new document,
|
||||
[Hands-on experiments with Machi and Humming Consensus](doc/humming-consensus-demo.md)
|
||||
is now available. It is a tutorial for setting up a 3 virtual
|
||||
machine Machi cluster and how to demonstrate the chain manager's
|
||||
reactions to server stops & starts, crashes & restarts, and pauses
|
||||
(simulated by `SIGSTOP` and `SIGCONT`).
|
||||
* The chain manager can still make suboptimal-but-safe choices for
|
||||
chain transitions when a server hangs/pauses temporarily.
|
||||
* Recent chain manager changes have made the instability window
|
||||
much shorter when the slow/paused server resumes execution.
|
||||
* Scott believes that a modest change to the chain manager's
|
||||
calculation of a new projection can reduce flapping in this (and
|
||||
many other cases) less likely. Currently, the new local
|
||||
projection is calculated using only local state (i.e., the chain
|
||||
manager's internal state + the fitness server's state).
|
||||
However, if the "latest" projection read from the public
|
||||
projection stores were also input to the new projection
|
||||
calculation function, then many obviously bad projections can be
|
||||
avoided without needing rounds of Humming Consensus to
|
||||
demonstrate that a bad projection is bad.
|
||||
|
||||
* FLU/data server process
|
||||
* All known correctness bugs have been fixed.
|
||||
* Performance has not yet been measured. Performance measurement
|
||||
and enhancements are scheduled to start in the middle of March 2016.
|
||||
(This will include a much-needed update to the `basho_bench` driver.)
|
||||
|
||||
* Access protocols and client libraries
|
||||
* The protocol used by both external clients and internally (instead
|
||||
of using Erlang's native message passing mechanisms) is based on
|
||||
Protocol Buffers.
|
||||
* (Machi PB protocol specification: ./src/machi.proto)[./src/machi.proto]
|
||||
* At the moment, the PB specification contains two protocols.
|
||||
Sometime in the near future, the spec will be split to separate
|
||||
the external client API (the "high" protocol) from the internal
|
||||
communication API (the "low" protocol).
|
||||
|
||||
* Recent conference talks about Machi
|
||||
* Erlang Factory San Francisco 2016
|
||||
[the slides and video recording](http://www.erlang-factory.com/sfbay2016/scott-lystig-fritchie)
|
||||
will be available a few weeks after the conference ends on March
|
||||
11, 2016.
|
||||
* Ricon 2015
|
||||
* [The slides](http://ricon.io/archive/2015/slides/Scott_Fritchie_Ricon_2015.pdf)
|
||||
* and the [video recording](https://www.youtube.com/watch?v=yR5kHL1bu1Q&index=13&list=PL9Jh2HsAWHxIc7Tt2M6xez_TOP21GBH6M)
|
||||
are now available.
|
||||
* If you would like to run the Humming Consensus code (with or without
|
||||
the network partition simulator) as described in the RICON 2015
|
||||
presentation, please see the
|
||||
[Humming Consensus demo doc](./doc/humming_consensus_demo.md).
|
||||
|
||||
<a name="sec4">
|
||||
## 4. Contributing to Machi's development
|
||||
|
||||
### 4.1 License
|
||||
|
||||
Basho Technologies, Inc. as committed to licensing all work for Machi
|
||||
under the
|
||||
[Apache Public License version 2](./LICENSE). All authors of source code
|
||||
and documentation who agree with these licensing terms are welcome to
|
||||
contribute their ideas in any form: suggested design or features,
|
||||
documentation, and source code.
|
||||
|
||||
Machi is still a very young project within Basho, with a small team of
|
||||
developers; please bear with us as we grow out of "toddler" stage into
|
||||
a more mature open source software project.
|
||||
We invite all contributors to review the
|
||||
[CONTRIBUTING.md](./CONTRIBUTING.md) document for guidelines for
|
||||
working with the Basho development team.
|
||||
|
||||
### 4.2 Development environment requirements
|
||||
|
||||
All development to date has been done with Erlang/OTP version 17 on OS
|
||||
X. The only known limitations for using R16 are minor type
|
||||
specification difference between R16 and 17, but we strongly suggest
|
||||
continuing development using version 17.
|
||||
|
||||
We also assume that you have the standard UNIX/Linux developer
|
||||
tool chain for C and C++ applications. Also, we assume
|
||||
that Git and GNU Make are available.
|
||||
The utility used to compile the Machi source code,
|
||||
`rebar`, is pre-compiled and included in the repo.
|
||||
For more details, please see the
|
||||
[Machi development environment prerequisites doc](./doc/dev-prerequisites.md).
|
||||
|
||||
Machi has a dependency on the
|
||||
[ELevelDB](https://github.com/basho/eleveldb) library. ELevelDB only
|
||||
supports UNIX/Linux OSes and 64-bit versions of Erlang/OTP only; we
|
||||
apologize to Windows-based and 32-bit-based Erlang developers for this
|
||||
restriction.
|
||||
|
||||
### 4.3 New protocols and features
|
||||
|
||||
If you'd like to work on a protocol such as Thrift, UBF,
|
||||
msgpack over UDP, or some other protocol, let us know by
|
||||
[opening an issue to discuss it](./issues/new).
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
* To Do list
|
||||
|
||||
** DONE remove the escript* stuff from machi_util.erl
|
||||
** DONE Add functions to manipulate 1-chain projections
|
||||
|
||||
- Add epoch ID = epoch number + checksum of projection!
|
||||
Done via compare() func.
|
||||
|
||||
** DONE Change all protocol ops to add epoch ID
|
||||
** DONE Add projection store to each FLU.
|
||||
|
||||
*** DONE What should the API look like? (borrow from chain mgr PoC?)
|
||||
|
||||
Yeah, I think that's pretty complete. Steal it now, worry later.
|
||||
|
||||
*** DONE Choose protocol & TCP port. Share with get/put? Separate?
|
||||
|
||||
Hrm, I like the idea of having a single TCP port to talk to any single
|
||||
FLU.
|
||||
|
||||
To make the protocol "easy" to hack, how about using the same basic
|
||||
method as append/write where there's a variable size blob. But we'll
|
||||
format that blob as a term_to_binary(). Then dispatch to a single
|
||||
func, and pattern match Erlang style in that func.
|
||||
|
||||
*** DONE Do it.
|
||||
|
||||
** DONE Finish OTP'izing the Chain Manager with FLU & proj store processes
|
||||
** DONE Eliminate the timeout exception for the client: just {error,timeout} ret
|
||||
** DONE Move prototype/chain-manager code to "top" of source tree
|
||||
*** DONE Preserve current test code (leave as-is? tiny changes?)
|
||||
*** DONE Make chain manager code flexible enough to run "real world" or "sim"
|
||||
** DONE Add projection wedging logic to each FLU.
|
||||
** DONE Implement real data repair, orchestrated by the chain manager
|
||||
** DONE Change all protocol ops to enforce the epoch ID
|
||||
|
||||
- Add no-wedging state to make testing easier?
|
||||
|
||||
|
||||
** DONE Adapt the projection-aware, CR-implementing client from demo-day
|
||||
** DONE Add major comment sections to the CR-impl client
|
||||
** DONE Simple basho_bench driver, put some unscientific chalk on the benchtop
|
||||
** TODO Create parallel PULSE test for basic API plus chain manager repair
|
||||
** DONE Add client-side vs. server-side checksum type, expand client API?
|
||||
** TODO Add gproc and get rid of registered name rendezvous
|
||||
*** TODO Fixes the atom table leak
|
||||
*** TODO Fixes the problem of having active sequencer for the same prefix
|
||||
on two FLUS in the same VM
|
||||
|
||||
** TODO Fix all known bugs/cruft with Chain Manager (list below)
|
||||
*** DONE Fix known bugs
|
||||
*** DONE Clean up crufty TODO comments and other obvious cruft
|
||||
*** TODO Re-add verification step of stable epochs, including inner projections!
|
||||
*** TODO Attempt to remove cruft items in flapping_i?
|
||||
|
||||
** TODO Move the FLU server to gen_server behavior?
|
||||
|
||||
|
||||
* DONE Chain manager CP mode, Plan B
|
||||
** SKIP Maybe? Change ch_mgr to use middleworker
|
||||
**** DONE Is it worthwhile? Is the parallelism so important? No, probably.
|
||||
**** SKIP Move middleworker func to utility module?
|
||||
** DONE Add new proc to psup group
|
||||
*** DONE Name: machi_fitness
|
||||
** DONE ch_mgr keeps its current proc struct: i.e. same 1 proc as today
|
||||
** NO chmgr asks hosed mgr for hosed list @ start of react_to_env
|
||||
** DONE For all hosed, do *async*: try to read latest proj.
|
||||
*** NO If OK, inform hosed mgr: status change will be used by next HC iter.
|
||||
*** NO If fail, no change, because that server is already known to be hosed
|
||||
*** DONE For all non-hosed, continue as the chain manager code does today
|
||||
*** DONE Any new errors are added to UpNodes/DownNodes tracking as used today
|
||||
*** DONE At end of react loop, if UpNodes list differs, inform hosed mgr.
|
||||
|
||||
* DONE fitness_mon, the fitness monitor
|
||||
** DONE Map key & val sketch
|
||||
|
||||
Logical sketch:
|
||||
|
||||
Map key: ObservingServerName::atom()
|
||||
|
||||
Map val: { ObservingServerLastModTime::now(),
|
||||
UnfitList::list(ServerName::atom()),
|
||||
AdminDownList::list(ServerName::atom()),
|
||||
Props::proplist() }
|
||||
|
||||
Implementation sketch:
|
||||
|
||||
1. Use CRDT map.
|
||||
2. If map key is not atom, then atom->string or atom->binary is fine.
|
||||
3. For map value, is it possible CRDT LWW type?
|
||||
|
||||
** DONE Investigate riak_dt data structure definition, manipulating, etc.
|
||||
** DONE Add dependency on riak_dt
|
||||
** DONE Update is an entire dict from Observer O
|
||||
*** DONE Merge my pending map + update map + my last mod time + my unfit list
|
||||
*** DONE if merged /= pending:
|
||||
**** DONE Schedule async tick (more)
|
||||
|
||||
Tick message contains list of servers with differing state as of this
|
||||
instant in time... we want to avoid triggering decisions about
|
||||
fitness/unfitness for other servers where we might have received less
|
||||
than a full time period's worth of waiting.
|
||||
|
||||
**** DONE Spam merged map to All_list -- [Me]
|
||||
**** DONE Set pending <- merged
|
||||
|
||||
*** DONE When we receive an async tick
|
||||
**** DONE set active map <- pending map for all servers in ticks list
|
||||
**** DONE Send ch_mgr a react_to_env tick trigger
|
||||
*** DONE react_to_env tick trigger actions
|
||||
**** DONE Filter active map to remove stale entries (i.e. no update in 1 hour)
|
||||
**** DONE If time since last map spam is too long, spam our *pending* map
|
||||
**** DONE Proceed with normal react processing, using *active* map for AllHosed!
|
||||
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
### The auto-generated code of machi_pb.beam has some complaints, not fixed yet.
|
||||
machi_pb.erl:0:
|
||||
##################################################
|
||||
######## Specific types #####################
|
||||
##################################################
|
||||
Unknown types:
|
||||
basho_bench_config:get/2
|
||||
machi_partition_simulator:get/1
|
||||
hamcrest:matchspec/0
|
||||
##################################################
|
||||
######## Specific messages #####################
|
||||
##################################################
|
||||
machi_chain_manager1.erl:2473: The created fun has no local return
|
||||
machi_chain_manager1.erl:2184: The pattern <_P1, P2, Else = {'expected_author2', UPI1_tail, _}> can never match the type <#projection_v1{epoch_number::'undefined' | non_neg_integer(),epoch_csum::'undefined' | binary(),author_server::atom(),chain_name::atom(),all_members::'undefined' | [atom()],witnesses::[atom()],creation_time::'undefined' | {non_neg_integer(),non_neg_integer(),non_neg_integer()},mode::'ap_mode' | 'cp_mode',upi::'undefined' | [atom()],repairing::'undefined' | [atom()],down::'undefined' | [atom()],dbg::'undefined' | [any()],dbg2::'undefined' | [any()],members_dict::'undefined' | [{_,_}]},#projection_v1{epoch_number::'undefined' | non_neg_integer(),epoch_csum::binary(),author_server::atom(),chain_name::atom(),all_members::'undefined' | [atom()],witnesses::[atom()],creation_time::'undefined' | {non_neg_integer(),non_neg_integer(),non_neg_integer()},mode::'ap_mode' | 'cp_mode',upi::'undefined' | [atom()],repairing::'undefined' | [atom()],down::'undefined' | [atom()],dbg::'undefined' | [any()],dbg2::'undefined' | [any()],members_dict::'undefined' | [{_,_}]},'true'>
|
||||
machi_chain_manager1.erl:2233: The pattern <_P1 = {'projection_v1', _, _, _, _, _, _, _, 'cp_mode', UPI1, Repairing1, _, _, _, _}, _P2 = {'projection_v1', _, _, _, _, _, _, _, 'cp_mode', UPI2, Repairing2, _, _, _, _}, Else = {'epoch_not_si', EpochX, 'not_gt', EpochY}> can never match the type <#projection_v1{epoch_number::'undefined' | non_neg_integer(),epoch_csum::'undefined' | binary(),author_server::atom(),chain_name::atom(),all_members::'undefined' | [atom()],witnesses::[atom()],creation_time::'undefined' | {non_neg_integer(),non_neg_integer(),non_neg_integer()},mode::'ap_mode' | 'cp_mode',upi::'undefined' | [atom()],repairing::'undefined' | [atom()],down::'undefined' | [atom()],dbg::'undefined' | [any()],dbg2::'undefined' | [any()],members_dict::'undefined' | [{_,_}]},#projection_v1{epoch_number::'undefined' | non_neg_integer(),epoch_csum::binary(),author_server::atom(),chain_name::atom(),all_members::'undefined' | [atom()],witnesses::[atom()],creation_time::'undefined' | {non_neg_integer(),non_neg_integer(),non_neg_integer()},mode::'ap_mode' | 'cp_mode',upi::'undefined' | [atom()],repairing::'undefined' | [atom()],down::'undefined' | [atom()],dbg::'undefined' | [any()],dbg2::'undefined' | [any()],members_dict::'undefined' | [{_,_}]},'true'>
|
||||
|
|
@ -1,74 +0,0 @@
|
|||
## Machi Documentation Overview
|
||||
|
||||
For a Web-browsable version of a snapshot of the source doc "EDoc"
|
||||
Erlang documentation, please use this link:
|
||||
[Machi EDoc snapshot](https://basho.github.io/machi/edoc/).
|
||||
|
||||
## Documents in this directory
|
||||
|
||||
### high-level-machi.pdf
|
||||
|
||||
[high-level-machi.pdf](high-level-machi.pdf)
|
||||
is an overview of the high level design for
|
||||
Machi. Its abstract:
|
||||
|
||||
> Our goal is a robust & reliable, distributed, highly available large
|
||||
> file store based upon write-once registers, append-only files, Chain
|
||||
> Replication, and client-server style architecture. All members of
|
||||
> the cluster store all of the files. Distributed load
|
||||
> balancing/sharding of files is outside of the scope of this system.
|
||||
> However, it is a high priority that this system be able to integrate
|
||||
> easily into systems that do provide distributed load balancing,
|
||||
> e.g., Riak Core. Although strong consistency is a major feature of
|
||||
> Chain Replication, this document will focus mainly on eventual
|
||||
> consistency features --- strong consistency design will be discussed
|
||||
> in a separate document.
|
||||
|
||||
### high-level-chain-mgr.pdf
|
||||
|
||||
[high-level-chain-mgr.pdf](high-level-chain-mgr.pdf)
|
||||
is an overview of the techniques used by
|
||||
Machi to manage Chain Replication metadata state. It also provides an
|
||||
introduction to the Humming Consensus algorithm. Its abstract:
|
||||
|
||||
> Machi is an immutable file store, now in active development by Basho
|
||||
> Japan KK. Machi uses Chain Replication to maintain strong consistency
|
||||
> of file updates to all replica servers in a Machi cluster. Chain
|
||||
> Replication is a variation of primary/backup replication where the
|
||||
> order of updates between the primary server and each of the backup
|
||||
> servers is strictly ordered into a single "chain". Management of
|
||||
> Chain Replication's metadata, e.g., "What is the current order of
|
||||
> servers in the chain?", remains an open research problem. The
|
||||
> current state of the art for Chain Replication metadata management
|
||||
> relies on an external oracle (e.g., ZooKeeper) or the Elastic
|
||||
> Replication algorithm.
|
||||
>
|
||||
> This document describes the Machi chain manager, the component
|
||||
> responsible for managing Chain Replication metadata state. The chain
|
||||
> manager uses a new technique, based on a variation of CORFU, called
|
||||
> "humming consensus".
|
||||
> Humming consensus does not require active participation by all or even
|
||||
> a majority of participants to make decisions. Machi's chain manager
|
||||
> bases its logic on humming consensus to make decisions about how to
|
||||
> react to changes in its environment, e.g. server crashes, network
|
||||
> partitions, and changes by Machi cluster admnistrators. Once a
|
||||
> decision is made during a virtual time epoch, humming consensus will
|
||||
> eventually discover if other participants have made a different
|
||||
> decision during that epoch. When a differing decision is discovered,
|
||||
> new time epochs are proposed in which a new consensus is reached and
|
||||
> disseminated to all available participants.
|
||||
|
||||
### chain-self-management-sketch.org
|
||||
|
||||
[chain-self-management-sketch.org](chain-self-management-sketch.org)
|
||||
is a mostly-deprecated draft of
|
||||
an introduction to the
|
||||
self-management algorithm proposed for Machi. Most material has been
|
||||
moved to the [high-level-chain-mgr.pdf](high-level-chain-mgr.pdf) document.
|
||||
|
||||
### cluster (directory)
|
||||
|
||||
This directory contains the sketch of the cluster design
|
||||
strawman for partitioning/distributing/sharding files across a large
|
||||
number of independent Machi chains.
|
||||
|
||||
|
|
@ -1,258 +0,0 @@
|
|||
|
||||
# Using basho_bench to twiddle with Machi
|
||||
|
||||
"Twiddle"? Really, is that a word? (Yes, it is a real English word.)
|
||||
|
||||
## Benchmarking Machi's performance ... no, don't do it.
|
||||
|
||||
Machi isn't ready for benchmark testing. Its public-facing API isn't
|
||||
finished yet. Its internal APIs aren't quite finished yet either. So
|
||||
any results of "benchmarking" effort is something that has even less
|
||||
value **N** months from now than the usual benchmarking effort.
|
||||
|
||||
However, there are uses for a benchmark tool. For example, one of my
|
||||
favorites is to put **stress** on a system. I don't care about
|
||||
average or 99th-percentile latencies, but I **might** care very much
|
||||
about behavior.
|
||||
|
||||
* What happens if a Machi system is under moderate load, and then I
|
||||
stop one of the servers? What happens?
|
||||
* How quickly do the chain managers react?
|
||||
* How quickly do the client libraries within Machi react?
|
||||
* How quickly do the external client API libraries react?
|
||||
|
||||
* What happens if a Machi system is under heavy load, for example,
|
||||
100% CPU load. Not all 100% might be the Machi services. Some CPU
|
||||
consumption might be from the load generator, like `basho_bench`
|
||||
itself that is running on the same machine as a Machi server. Or
|
||||
perhaps it's a tiny C program that I wrote:
|
||||
|
||||
main()
|
||||
{ while (1) { ; } }
|
||||
|
||||
## An example of how adding moderate stress can find weird bugs
|
||||
|
||||
The driver/plug-in module for `basho_bench` is only a few hours old.
|
||||
(I'm writing on Wednesday, 2015-05-20.) But just now, I configured my
|
||||
basho_bench config file to try to contact a Machi cluster of three
|
||||
nodes ... but really, only one was running. The client library,
|
||||
`machi_cr_client.erl`, has **an extremely simple** method for dealing
|
||||
with failed servers. I know it's simple and dumb, but that's OK in
|
||||
many cases.
|
||||
|
||||
However, `basho_bench` and the `machi_cr_client.erl` were acting very,
|
||||
very badly. I couldn't figure it out until I took a peek at my OS's
|
||||
`dmesg` output, namely: `dmesg | tail`. It said things like this:
|
||||
|
||||
Limiting closed port RST response from 690 to 50 packets per second
|
||||
Limiting closed port RST response from 367 to 50 packets per second
|
||||
Limiting closed port RST response from 101 to 50 packets per second
|
||||
Limiting closed port RST response from 682 to 50 packets per second
|
||||
Limiting closed port RST response from 467 to 50 packets per second
|
||||
|
||||
Well, isn't that interesting?
|
||||
|
||||
This system was running on a single OS X machine: my MacBook Pro
|
||||
laptop, running OS X 10.10 (Yosemite). I have seen that error
|
||||
before. And I know how to fix it.
|
||||
|
||||
* **Option 1**: Change the client library config to ignore the Machi
|
||||
servers that I know will always be down during my experiment.
|
||||
* ** Option 2**: Use the following to change my OS's TCP stack RST
|
||||
behavior. (If a TCP port is not being listened to, the OS will
|
||||
send a RST packet to signal "connection refused".)
|
||||
|
||||
On OS X, the limit for RST packets is 50/second. The
|
||||
`machi_cr_client.erl` client can generate far more than 50/second, as
|
||||
the `Limiting closed port RST response...` messages above show. So, I
|
||||
used some brute-force to change the environment:
|
||||
|
||||
sudo sysctl -w net.inet.icmp.icmplim=20000
|
||||
|
||||
... and the problem disappeared.
|
||||
|
||||
## Starting with basho_bench: a step-by-step tutorial
|
||||
|
||||
First, clone the `basho_bench` source code, then compile it. You will
|
||||
need Erlang/OTP version R16B or later to compile. I recommend using
|
||||
Erlang/OTP 17.x, because I've been doing my Machi development using
|
||||
17.x.
|
||||
|
||||
cd /some/nice/dev/place
|
||||
git clone https://github.com/basho/basho_bench.git
|
||||
cd basho_bench
|
||||
make
|
||||
|
||||
In order to create graphs of `basho_bench` output, you'll need
|
||||
installed one of the following:
|
||||
|
||||
* R (the statistics package)
|
||||
* gnuplot
|
||||
|
||||
If you don't have either available on the machine(s) you're testing,
|
||||
but you do have R (or gnuplot) on some other machine **Y**, then you can
|
||||
copy the output files to machine **Y** and generate the graphs there.
|
||||
|
||||
## Compiling the Machi source
|
||||
|
||||
First, clone the Machi source code, then compile it. You will
|
||||
need Erlang/OTP version 17.x to compile.
|
||||
|
||||
cd /some/nice/dev/place
|
||||
git clone https://github.com/basho/machi.git
|
||||
cd machi
|
||||
make
|
||||
|
||||
## Creating a basho_bench test configuration file.
|
||||
|
||||
There are a couple of example `basho_bench` configuration files in the
|
||||
Machi `priv` directory.
|
||||
|
||||
* [basho_bench.append-example.config](priv/basho_bench.append-example.config),
|
||||
an example for writing Machi files.
|
||||
* [basho_bench.read-example.config](priv/basho_bench.read-example.config),
|
||||
an example for reading Machi files.
|
||||
|
||||
If you want a test to do both reading & writing ... well, the
|
||||
driver/plug-in is not mature enough to do it **well**. If you really
|
||||
want to, refer to the `basho_bench` docs for how to use the
|
||||
`operations` config option.
|
||||
|
||||
The `basho_bench` config file is configured in Erlang term format.
|
||||
Each configuration item is a 2-tuple followed by a period. Comments
|
||||
begin with a `%` character and continue to the end-of-line.
|
||||
|
||||
%% Mandatory: adjust this code path to top of your compiled Machi source distro
|
||||
{code_paths, ["/Users/fritchie/b/src/machi"]}.
|
||||
{driver, machi_basho_bench_driver}.
|
||||
|
||||
%% Chose your maximum rate (per worker proc, see 'concurrent' below)
|
||||
{mode, {rate, 25}}.
|
||||
|
||||
%% Runtime & reporting interval
|
||||
{duration, 10}. % minutes
|
||||
{report_interval, 1}. % seconds
|
||||
|
||||
%% Choose your number of worker procs
|
||||
{concurrent, 5}.
|
||||
|
||||
%% Here's a chain of (up to) length 3, all on localhost
|
||||
{machi_server_info,
|
||||
[
|
||||
{p_srvr,a,machi_flu1_client,"localhost",4444,[]},
|
||||
{p_srvr,b,machi_flu1_client,"localhost",4445,[]},
|
||||
{p_srvr,c,machi_flu1_client,"localhost",4446,[]}
|
||||
]}.
|
||||
{machi_ets_key_tab_type, set}. % 'set' or 'ordered_set'
|
||||
|
||||
%% Workload-specific definitions follow....
|
||||
|
||||
%% 10 parts 'append' operation + 0 parts anything else = 100% 'append' ops
|
||||
{operations, [{append, 10}]}.
|
||||
|
||||
%% For append, key = Machi file prefix name
|
||||
{key_generator, {concat_binary, <<"prefix">>,
|
||||
{to_binstr, "~w", {uniform_int, 30}}}}.
|
||||
|
||||
%% Increase size of value_generator_source_size if value_generator is big!!
|
||||
{value_generator_source_size, 2111000}.
|
||||
{value_generator, {fixed_bin, 32768}}. % 32 KB
|
||||
|
||||
In summary:
|
||||
|
||||
* Yes, you really need to change `code_paths` to be the same as your
|
||||
`/some/nice/dev/place/basho_bench` directory ... and that directory
|
||||
must be on the same machine(s) that you intend to run `basho_bench`.
|
||||
* Each worker process will have a rate limit of 25 ops/sec.
|
||||
* The test will run for 10 minutes and report stats every 1 second.
|
||||
* There are 5 concurrent worker processes. Each worker will
|
||||
concurrently issue commands from the `operations` list, within the
|
||||
workload throttle limit.
|
||||
* The Machi cluster is a collection of three servers, all on
|
||||
"localhost", and using TCP ports 4444-4446.
|
||||
* Don't change the `machi_ets_key_tab_type`
|
||||
* Our workload operation mix is 100% `append` operations.
|
||||
* The key generator for the `append` operation specifies the file
|
||||
prefix that will be chosen (at pseudo-random). In this case, we'll
|
||||
choose uniformly randomly between file prefix `prefix0` and
|
||||
`prefix29`.
|
||||
* The values that we append will be fixed 32KB length, but they will
|
||||
be chosen from a random byte string of 2,111,000 bytes.
|
||||
|
||||
There are many other options for `basho_bench`, especially for the
|
||||
`key_generator` and `value_generator` options. Please see the
|
||||
`basho_bench` docs for further information.
|
||||
|
||||
## Running basho_bench
|
||||
|
||||
You can run `basho_bench` using the command:
|
||||
|
||||
/some/nice/dev/place/basho_bench/basho_bench /path/to/config/file
|
||||
|
||||
... where `/path/to/config/file` is the path to your config file. (If
|
||||
you use an example from the `priv` dir, we recommend that you make a
|
||||
copy elsewhere, edit the copy, and then use the copy to run
|
||||
`basho_bench`.)
|
||||
|
||||
You'll create a stats output directory, called `tests`, in the current
|
||||
working directory. (Add `{results_dir, "/some/output/dir"}.` to change
|
||||
the default!)
|
||||
|
||||
Each time `basho_bench` is run, a new output stats directory is
|
||||
created in the `tests` directory. The symbolic link `tests/current`
|
||||
will always point to the last `basho_bench` run's output. But all
|
||||
prior results are always accessible! Take a look in this directory
|
||||
for all of the output.
|
||||
|
||||
## Generating some pretty graphs
|
||||
|
||||
If you are using R, then the following command will create a graph:
|
||||
|
||||
Rscript --vanilla /some/nice/dev/place/basho_bench/basho_bench/priv/summary.r -i $CWD/tests/current
|
||||
|
||||
If the `tests` directory is not in your current working dir (i.e. not
|
||||
in `$CWD`), then please alter the command accordingly.
|
||||
|
||||
R will create the final results graph in `$CWD/tests/current/summary.png`.
|
||||
|
||||
If you are using gnuplot, please look at
|
||||
`/some/nice/dev/place/basho_bench/basho_bench/Makefile` to see how to
|
||||
use gnuplot to create the final results graph.
|
||||
|
||||
## An example graph
|
||||
|
||||
So, without a lot of context about the **Machi system** or about the
|
||||
**basho_bench system** or about the ops being performed, here is an
|
||||
example graph that was created by R:
|
||||
|
||||
|
||||
|
||||
**Without context??* How do I remember the context?
|
||||
|
||||
My recommendation is: always keep the `.config` file together with the
|
||||
graph file. In the `tests` directory, `basho_bench` will always make
|
||||
a copy of the config file used to generate the test data.
|
||||
|
||||
This config tells you very little about the environment of the load
|
||||
generator machine or the Machi cluster, but ... you need to maintain
|
||||
that documentation yourself, please! You'll thank me for that advice,
|
||||
someday, 11 months from now when you can't remember the details of
|
||||
that important test that you ran so very long ago.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Really, we don't recommend using `basho_bench` for any serious
|
||||
performance measurement of Machi yet: Machi needs more maturity before
|
||||
it's reasonable to measure & judge its performance. But stress
|
||||
testing is indeed useful for reasons other than measuring
|
||||
Nth-percentile latency of operation `flarfbnitz`. We hope that this
|
||||
tutorial has been helpful!
|
||||
|
||||
If you encounter any difficulty with this tutorial or with Machi,
|
||||
please open an issue/ticket at [GH Issues for
|
||||
Machi](https://github.com/basho/machi/issues) ... use the green "New
|
||||
issue" button. There are bugs and misfeatures in the `basho_bench`
|
||||
plugin, sorry, but please help us fix them.
|
||||
|
||||
> -Scott Lystig Fritchie,
|
||||
> Machi Team @ Basho
|
||||
Binary file not shown.
Binary file not shown.
|
|
@ -1,251 +0,0 @@
|
|||
-*- mode: org; -*-
|
||||
#+TITLE: Machi Chain Self-Management Sketch
|
||||
#+AUTHOR: Scott
|
||||
#+STARTUP: lognotedone hidestars indent showall inlineimages
|
||||
#+SEQ_TODO: TODO WORKING WAITING DONE
|
||||
|
||||
* 1. Abstract
|
||||
The high level design of the Machi "chain manager" has moved to the
|
||||
[[high-level-chain-manager.pdf][Machi chain manager high level design]] document.
|
||||
|
||||
We try to discuss the network partition simulator that the
|
||||
algorithm runs in and how the algorithm behaves in both symmetric and
|
||||
asymmetric network partition scenarios. The symmetric partition cases
|
||||
are all working well (surprising in a good way), and the asymmetric
|
||||
partition cases are working well (in a damn mystifying kind of way).
|
||||
It'd be really, *really* great to get more review of the algorithm and
|
||||
the simulator.
|
||||
|
||||
* 2. Copyright
|
||||
|
||||
#+BEGIN_SRC
|
||||
%% Copyright (c) 2015 Basho Technologies, Inc. All Rights Reserved.
|
||||
%%
|
||||
%% This file is provided to you under the Apache License,
|
||||
%% Version 2.0 (the "License"); you may not use this file
|
||||
%% except in compliance with the License. You may obtain
|
||||
%% a copy of the License at
|
||||
%%
|
||||
%% http://www.apache.org/licenses/LICENSE-2.0
|
||||
%%
|
||||
%% Unless required by applicable law or agreed to in writing,
|
||||
%% software distributed under the License is distributed on an
|
||||
%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
%% KIND, either express or implied. See the License for the
|
||||
%% specific language governing permissions and limitations
|
||||
%% under the License.
|
||||
#+END_SRC
|
||||
|
||||
|
||||
* 3. Document restructuring
|
||||
|
||||
Much of the text previously appearing in this document has moved to the
|
||||
[[high-level-chain-manager.pdf][Machi chain manager high level design]] document.
|
||||
|
||||
* 4. Diagram of the self-management algorithm
|
||||
|
||||
** WARNING: This section is now deprecated
|
||||
|
||||
The definitive text for this section has moved to the [[high-level-chain-manager.pdf][Machi chain
|
||||
manager high level design]] document.
|
||||
|
||||
** Flowchart notes
|
||||
|
||||
*** Algorithm execution rates / sleep intervals between executions
|
||||
|
||||
Due to the ranking algorithm's preference for author node names that
|
||||
are large (lexicographically), nodes with larger node names should
|
||||
execute the algorithm more frequently than other nodes. The reason
|
||||
for this is to try to avoid churn: a proposal by a "small" node may
|
||||
propose a UPI list of L at epoch 10, and a few moments later a "big"
|
||||
node may propose the same UPI list L at epoch 11. In this case, there
|
||||
would be two chain state transitions: the epoch 11 projection would be
|
||||
ranked higher than epoch 10's projection. If the "big" node
|
||||
executed more frequently than the "small" node, then it's more likely
|
||||
that epoch 10 would be written by the "big" node, which would then
|
||||
cause the "small" node to stop at state A40 and avoid any
|
||||
externally-visible action.
|
||||
|
||||
*** A simple example race between two participants noting a 3rd's failure
|
||||
|
||||
Assume a chain of three nodes, A, B, and C. In a projection at epoch
|
||||
E. For all nodes, the P_current projection at epoch E is:
|
||||
|
||||
#+BEGIN_QUOTE
|
||||
UPI=[A,B,C], Repairing=[], Down=[]
|
||||
#+END_QUOTE
|
||||
|
||||
Now assume that C crashes during epoch E. The failure detector
|
||||
running locally at both A & B eventually notice C's death. The new
|
||||
information triggers a new iteration of the self-management algorithm.
|
||||
A calculates its P_newprop (call it P_newprop_a) and writes it to its
|
||||
own public projection store. Meanwhile, B does the same and wins the
|
||||
race to write P_newprop_b to its own public projection store.
|
||||
|
||||
At this instant in time, the public projection stores of each node
|
||||
looks something like this:
|
||||
|
||||
|-------+--------------+--------------+--------------|
|
||||
| Epoch | Node A | Node B | Node C |
|
||||
|-------+--------------+--------------+--------------|
|
||||
| E | UPI=[A,B,C] | UPI=[A,B,C] | UPI=[A,B,C] |
|
||||
| | Repairing=[] | Repairing=[] | Repairing=[] |
|
||||
| | Down=[] | Down=[] | Down=[] |
|
||||
| | Author=A | Author=A | Author=A |
|
||||
|-------+--------------+--------------+--------------|
|
||||
| E+1 | UPI=[A,B] | UPI=[A,B] | C is dead, |
|
||||
| | Repairing=[] | Repairing=[] | unwritten |
|
||||
| | Down=[C] | Down=[C] | |
|
||||
| | Author=A | Author=B | |
|
||||
|-------+--------------+--------------+--------------|
|
||||
|
||||
If we use the CORFU-style projection naming convention, where a
|
||||
projection's name is exactly equal to the epoch number, then all
|
||||
participants cannot tell the difference between the projection at
|
||||
epoch E+1 authored by node A from the projection at epoch E+1 authored
|
||||
by node B: the names are the same, i.e., E+1.
|
||||
|
||||
Machi must extend the original CORFU protocols by changing the name of
|
||||
the projection. In Machi's case, the projection is named by this
|
||||
2-tuple:
|
||||
#+BEGIN_SRC
|
||||
{epoch #, hash of the entire projection (minus hash field itself)}
|
||||
#+END_SRC
|
||||
|
||||
This name is used in all relevant APIs where the name is required to
|
||||
make a wedge state transition. In the case of the example & table
|
||||
above, all of the UPI & Repairing & Down lists are equal. However, A
|
||||
& B's unanimity is due to the symmetric nature of C's partition: C is
|
||||
dead. In the case of an asymmetric partition of C, it is indeed
|
||||
possible for A's version of epoch E+1's UPI list to be different from
|
||||
B's UPI list in the same epoch E+1.
|
||||
|
||||
*** A second example, building on the first example
|
||||
|
||||
Building on the first example, let's assume that A & B have reconciled
|
||||
their proposals for epoch E+2. Nodes A & B are running under a
|
||||
unanimous proposal at E+2.
|
||||
|
||||
|-------+--------------+--------------+--------------|
|
||||
| E+2 | UPI=[A,B] | UPI=[A,B] | C is dead, |
|
||||
| | Repairing=[] | Repairing=[] | unwritten |
|
||||
| | Down=[C] | Down=[C] | |
|
||||
| | Author=A | Author=A | |
|
||||
|-------+--------------+--------------+--------------|
|
||||
|
||||
Now assume that C restarts. It was dead for a little while, and its
|
||||
code is slightly buggy. Node C decides to make a proposal without
|
||||
first consulting its failure detector: let's assume that C believes
|
||||
that only C is alive. Also, C knows that epoch E was the last epoch
|
||||
valid before it crashed, so it decides that it will write its new
|
||||
proposal at E+2. The result is a set of public projection stores that
|
||||
look like this:
|
||||
|
||||
|-----+--------------+--------------+--------------|
|
||||
| E+2 | UPI=[A,B] | UPI=[A,B] | UPI=[C] |
|
||||
| | Repairing=[] | Repairing=[] | Repairing=[] |
|
||||
| | Down=[C] | Down=[C] | Down=[A,B] |
|
||||
| | Author=A | Author=A | Author=C |
|
||||
|-----+--------------+--------------+--------------|
|
||||
|
||||
Now we're in a pickle where a client C could read the latest
|
||||
projection from node C and get a different view of the world than if
|
||||
it had read the latest projection from nodes A or B.
|
||||
|
||||
If running in AP mode, this wouldn't be a big problem: a write to node
|
||||
C only (or a write to nodes A & B only) would be reconciled
|
||||
eventually. Also, eventually, one of the nodes would realize that C
|
||||
was no longer partitioned and would make a new proposal at epoch E+3.
|
||||
|
||||
If running in CP mode, then any client that attempted to use C's
|
||||
version of the E+2 projection would fail: the UPI list does not
|
||||
contain a quorum majority of nodes. (Other discussion of CP mode's
|
||||
use of quorum majority for UPI members is out of scope of this
|
||||
document. Also out of scope is the use of "witness servers" to
|
||||
augment the quorum majority UPI scheme.)
|
||||
|
||||
* 5. The Network Partition Simulator
|
||||
** Overview
|
||||
The function machi_chain_manager1_test:convergence_demo_test()
|
||||
executes the following in a simulated network environment within a
|
||||
single Erlang VM:
|
||||
|
||||
#+BEGIN_QUOTE
|
||||
Test the convergence behavior of the chain self-management algorithm
|
||||
for Machi.
|
||||
|
||||
1. Set up 4 FLUs and chain manager pairs.
|
||||
|
||||
2. Create a number of different network partition scenarios, where
|
||||
(simulated) partitions may be symmetric or asymmetric. (At the
|
||||
Seattle 2015 meet-up, I called this the "shaking the snow globe"
|
||||
phase, where asymmetric network partitions are simulated and are
|
||||
calculated at random differently for each simulated node. During
|
||||
this time, the simulated network is wildly unstable.)
|
||||
|
||||
3. Then halt changing the partitions and keep the simulated network
|
||||
stable. The simulated may remain broken (i.e. at least one
|
||||
asymmetric partition remains in effect), but at least it's
|
||||
stable.
|
||||
|
||||
4. Run a number of iterations of the algorithm in parallel by poking
|
||||
each of the manager processes on a random'ish basis to simulate
|
||||
the passage of time.
|
||||
|
||||
5. Afterward, fetch the chain transition histories made by each FLU
|
||||
and verify that no transition was ever unsafe.
|
||||
#+END_QUOTE
|
||||
|
||||
|
||||
** Behavior in symmetric network partitions
|
||||
|
||||
The simulator has yet to find an error. This is both really cool and
|
||||
really terrifying: is this *really* working? No, seriously, where are
|
||||
the bugs? Good question. Both the algorithm and the simulator need
|
||||
review and futher study.
|
||||
|
||||
In fact, it'd be awesome if I could work with someone who has more
|
||||
TLA+ experience than I do to work on a formal specification of the
|
||||
self-management algorithm and verify its correctness.
|
||||
|
||||
** Behavior in asymmetric network partitions
|
||||
|
||||
Text has moved to the [[high-level-chain-manager.pdf][Machi chain manager high level design]] document.
|
||||
|
||||
* Prototype notes
|
||||
|
||||
** Mid-April 2015
|
||||
|
||||
I've finished moving the chain manager plus the inner/nested
|
||||
projection code into the top-level 'src' dir of this repo. The idea
|
||||
is working very well under simulation, more than well enough to gamble
|
||||
on for initial use.
|
||||
|
||||
Stronger validation work will continue through 2015, ideally using a
|
||||
tool like TLA+.
|
||||
|
||||
** Mid-March 2015
|
||||
|
||||
I've come to realize that the property that causes the nice property
|
||||
of "Were my last 2L proposals identical?" also requires that the
|
||||
proposals be *stable*. If a participant notices, "Hey, there's
|
||||
flapping happening, so I'll propose a different projection
|
||||
P_different", then the very act of proposing P_different disrupts the
|
||||
"last 2L proposals identical" cycle the enables us to detect
|
||||
flapping. We kill the goose that's laying our golden egg.
|
||||
|
||||
I've been working on the idea of "nested" projections, namely an
|
||||
"outer" and "inner" projection. Only the "outer projection" is used
|
||||
for cycle detection. The "inner projection" is the same as the outer
|
||||
projection when flapping is not detected. When flapping is detected,
|
||||
then the inner projection is one that excludes all nodes that the
|
||||
outer projection has identified as victims of asymmetric partition.
|
||||
|
||||
This inner projection technique may or may not work well enough to
|
||||
use? It would require constant flapping of the outer proposal, which
|
||||
is going to consume CPU and also chew up projection store keys with
|
||||
the flapping churn. That churn would continue as long as an
|
||||
asymmetric partition exists. The simplest way to cope with this would
|
||||
be to reduce proposal rates significantly, say 10x or 50x slower, to
|
||||
slow churn down to proposals from several-per-second to perhaps
|
||||
several-per-minute?
|
||||
|
|
@ -1,103 +0,0 @@
|
|||
#FIG 3.2 Produced by xfig version 3.2.5b
|
||||
Landscape
|
||||
Center
|
||||
Inches
|
||||
Letter
|
||||
94.00
|
||||
Single
|
||||
-2
|
||||
1200 2
|
||||
6 7425 2700 8700 3300
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 645 7425 2895 After\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 255 1215 7425 3210 Migration\001
|
||||
-6
|
||||
6 7425 450 8700 1050
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 780 7425 675 Before\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 255 1215 7425 990 Migration\001
|
||||
-6
|
||||
6 75 1425 6900 2325
|
||||
6 4875 1425 6900 2325
|
||||
6 5400 1575 6375 2175
|
||||
4 0 0 50 -1 2 14 0.0000 4 165 390 5400 1800 Not\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 225 945 5400 2100 migrated\001
|
||||
-6
|
||||
2 2 1 2 0 7 50 -1 -1 6.000 0 0 -1 0 0 5
|
||||
4950 1500 6825 1500 6825 2250 4950 2250 4950 1500
|
||||
-6
|
||||
6 2475 1425 4500 2325
|
||||
6 3000 1575 3975 2175
|
||||
4 0 0 50 -1 2 14 0.0000 4 165 390 3000 1800 Not\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 225 945 3000 2100 migrated\001
|
||||
-6
|
||||
2 2 1 2 0 7 50 -1 -1 6.000 0 0 -1 0 0 5
|
||||
2550 1500 4425 1500 4425 2250 2550 2250 2550 1500
|
||||
-6
|
||||
6 75 1425 2100 2325
|
||||
6 600 1575 1575 2175
|
||||
4 0 0 50 -1 2 14 0.0000 4 165 390 600 1800 Not\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 225 945 600 2100 migrated\001
|
||||
-6
|
||||
2 2 1 2 0 7 50 -1 -1 6.000 0 0 -1 0 0 5
|
||||
150 1500 2025 1500 2025 2250 150 2250 150 1500
|
||||
-6
|
||||
-6
|
||||
2 1 0 2 0 7 50 -1 -1 6.000 0 0 -1 1 0 2
|
||||
1 1 3.00 60.00 120.00
|
||||
150 4200 150 3750
|
||||
2 1 0 2 0 7 50 -1 -1 6.000 0 0 -1 1 0 2
|
||||
1 1 3.00 60.00 120.00
|
||||
3750 4200 3750 3750
|
||||
2 1 0 2 0 7 50 -1 -1 6.000 0 0 -1 1 0 2
|
||||
1 1 3.00 60.00 120.00
|
||||
2025 4200 2025 3750
|
||||
2 1 0 2 0 7 50 -1 -1 6.000 0 0 -1 1 0 2
|
||||
1 1 3.00 60.00 120.00
|
||||
7350 4200 7350 3750
|
||||
2 1 0 2 0 7 50 -1 -1 6.000 0 0 -1 1 0 2
|
||||
1 1 3.00 60.00 120.00
|
||||
5550 4200 5550 3750
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
2550 0 2550 1500 150 1500 150 0 2550 0
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
4950 0 4950 1500 2550 1500 2550 0 4950 0
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
7350 0 7350 1500 4950 1500 4950 0 7350 0
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
150 2250 2025 2250 2025 3750 150 3750 150 2250
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
4425 2250 4950 2250 4950 3750 4425 3750 4425 2250
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
4950 2250 6825 2250 6825 3750 4950 3750 4950 2250
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
6825 2250 7350 2250 7350 3750 6825 3750 6825 2250
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
2025 2250 2550 2250 2550 3750 2025 3750 2025 2250
|
||||
2 2 0 3 0 7 50 -1 -1 0.000 0 0 -1 0 0 5
|
||||
2550 2250 4425 2250 4425 3750 2550 3750 2550 2250
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 480 75 4500 0.00\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 480 6825 4500 1.00\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 480 1725 4500 0.25\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 480 3525 4500 0.50\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 195 480 5250 4500 0.75\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 450 1275 ~33% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 2925 1275 ~33% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 5250 1275 ~33% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 180 495 2025 3525 ~8%\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 300 3525 ~25% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 2625 3525 ~25% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 180 495 4425 3525 ~8%\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 240 1710 5025 3525 ~25% total keys\001
|
||||
4 0 0 50 -1 2 14 0.0000 4 180 495 6825 3525 ~8%\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 195 2175 3075 4\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 195 4575 3075 4\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 195 6975 3075 4\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 600 600 Chain1\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 3000 600 Chain2\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 5400 600 Chain3\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 285 2100 2625 C\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 285 4500 2625 C\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 285 6900 2625 C\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 525 2850 Chain1\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 2925 2850 Chain2\001
|
||||
4 0 0 50 -1 2 24 0.0000 4 270 1245 5325 2850 Chain3\001
|
||||
4 0 0 50 -1 2 18 0.0000 4 240 4350 1350 4875 Cluster locator, on the unit interval\001
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 7.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 7.4 KiB |
|
|
@ -1,481 +0,0 @@
|
|||
-*- mode: org; -*-
|
||||
#+TITLE: Machi cluster "name game" sketch
|
||||
#+AUTHOR: Scott
|
||||
#+STARTUP: lognotedone hidestars indent showall inlineimages
|
||||
#+SEQ_TODO: TODO WORKING WAITING DONE
|
||||
#+COMMENT: M-x visual-line-mode
|
||||
#+COMMENT: Also, disable auto-fill-mode
|
||||
|
||||
* 1. "Name Games" with random-slicing style consistent hashing
|
||||
|
||||
Our goal: to distribute lots of files very evenly across a large
|
||||
collection of individual, small Machi chains.
|
||||
|
||||
* 2. Assumptions
|
||||
|
||||
** Basic familiarity with Machi high level design and Machi's "projection"
|
||||
|
||||
The [[https://github.com/basho/machi/blob/master/doc/high-level-machi.pdf][Machi high level design document]] contains all of the basic
|
||||
background assumed by the rest of this document.
|
||||
|
||||
** Analogy: "neighborhood : city :: Machi chain : Machi cluster"
|
||||
|
||||
Analogy: The word "machi" in Japanese means small town or
|
||||
neighborhood. As the Tokyo Metropolitan Area is built from many
|
||||
machis and smaller cities, therefore a big, partitioned file store can
|
||||
be built out of many small Machi chains.
|
||||
|
||||
** Familiarity with the Machi chain concept
|
||||
|
||||
It's clear (I hope!) from
|
||||
the [[https://github.com/basho/machi/blob/master/doc/high-level-machi.pdf][Machi high level design document]] that Machi alone does not support
|
||||
any kind of file partitioning/distribution/sharding across multiple
|
||||
small Machi chains. There must be another layer above a Machi chain to
|
||||
provide such partitioning services.
|
||||
|
||||
Using the [[https://github.com/basho/machi/tree/master/prototype/demo-day-hack][cluster quick-and-dirty prototype]] as an
|
||||
architecture sketch, let's now assume that we have ~n~ independent Machi
|
||||
chains. We assume that each of these chains has the same
|
||||
chain length in the nominal case, e.g. chain length of 3.
|
||||
We wish to provide partitioned/distributed file storage
|
||||
across all ~n~ chains. We call the entire collection of ~n~ Machi
|
||||
chains a "cluster".
|
||||
|
||||
We may wish to have several types of Machi clusters. For example:
|
||||
|
||||
+ Chain length of 1 for "don't care if it gets lost,
|
||||
store stuff very very cheaply" data.
|
||||
+ Chain length of 2 for normal data.
|
||||
+ Equivalent to quorum replication's reliability with 3 copies.
|
||||
+ Chain length of 7 for critical, unreplaceable data.
|
||||
+ Equivalent to quorum replication's reliability with 15 copies.
|
||||
|
||||
Each of these types of chains will have a name ~N~ in the
|
||||
namespace. The role of the cluster namespace will be demonstrated in
|
||||
Section 3 below.
|
||||
|
||||
** Continue an early assumption: a Machi chain is unaware of clustering
|
||||
|
||||
Let's continue with an assumption that an individual Machi chain
|
||||
inside of a cluster is completely unaware of the cluster layer.
|
||||
|
||||
** The reader is familiar with the random slicing technique
|
||||
|
||||
I'd done something very-very-nearly-like-this for the Hibari database
|
||||
6 years ago. But the Hibari technique was based on stuff I did at
|
||||
Sendmail, Inc, in 2000, so this technique feels like old news to me.
|
||||
{shrug}
|
||||
|
||||
The following section provides an illustrated example.
|
||||
Very quickly, the random slicing algorithm is:
|
||||
|
||||
- Hash a string onto the unit interval [0.0, 1.0)
|
||||
- Calculate h(unit interval point, Map) -> bin, where ~Map~ divides
|
||||
the unit interval into bins (or partitions or shards).
|
||||
|
||||
Machi's adaptation is in step 1: we do not hash any strings. Instead, we
|
||||
simply choose a number on the unit interval. This number is called
|
||||
the "cluster locator number".
|
||||
|
||||
As described later in this doc, Machi file names are structured into
|
||||
several components. One component of the file name contains the cluster
|
||||
locator number; we use the number as-is for step 2 above.
|
||||
|
||||
*** For more information about Random Slicing
|
||||
|
||||
For a comprehensive description of random slicing, please see the
|
||||
first two papers. For a quicker summary, please see the third
|
||||
reference.
|
||||
|
||||
#+BEGIN_QUOTE
|
||||
Reliable and Randomized Data Distribution Strategies for Large Scale Storage Systems
|
||||
Alberto Miranda et al.
|
||||
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.226.5609
|
||||
(short version, HIPC'11)
|
||||
|
||||
Random Slicing: Efficient and Scalable Data Placement for Large-Scale
|
||||
Storage Systems
|
||||
Alberto Miranda et al.
|
||||
DOI: http://dx.doi.org/10.1145/2632230 (long version, ACM Transactions
|
||||
on Storage, Vol. 10, No. 3, Article 9, 2014)
|
||||
|
||||
[[http://hibari.github.io/hibari-doc/hibari-sysadmin-guide.en.html#chain-migration][Hibari Sysadmin Guide, chain migration section]].
|
||||
http://hibari.github.io/hibari-doc/hibari-sysadmin-guide.en.html#chain-migration
|
||||
#+END_QUOTE
|
||||
|
||||
* 3. A simple illustration
|
||||
|
||||
We use a variation of the Random Slicing hash that we will call
|
||||
~rs_hash_with_float()~. The Erlang-style function type is shown
|
||||
below.
|
||||
|
||||
#+BEGIN_SRC erlang
|
||||
%% type specs, Erlang-style
|
||||
-spec rs_hash_with_float(float(), rs_hash:map()) -> rs_hash:chain_id().
|
||||
#+END_SRC
|
||||
|
||||
I'm borrowing an illustration from the HibariDB documentation here,
|
||||
but it fits my purposes quite well. (I am the original creator of that
|
||||
image, and also the use license is compatible.)
|
||||
|
||||
#+CAPTION: Illustration of 'Map', using four Machi chains
|
||||
|
||||
[[./migration-4.png]]
|
||||
|
||||
Assume that we have a random slicing map called ~Map~. This particular
|
||||
~Map~ maps the unit interval onto 4 Machi chains:
|
||||
|
||||
| Hash range | Chain ID |
|
||||
|-------------+----------|
|
||||
| 0.00 - 0.25 | Chain1 |
|
||||
| 0.25 - 0.33 | Chain4 |
|
||||
| 0.33 - 0.58 | Chain2 |
|
||||
| 0.58 - 0.66 | Chain4 |
|
||||
| 0.66 - 0.91 | Chain3 |
|
||||
| 0.91 - 1.00 | Chain4 |
|
||||
|
||||
Assume that the system chooses a cluster locator of 0.05.
|
||||
According to ~Map~, the value of
|
||||
~rs_hash_with_float(0.05,Map) = Chain1~.
|
||||
Similarly, ~rs_hash_with_float(0.26,Map) = Chain4~.
|
||||
|
||||
This example should look very similar to Hibari's technique.
|
||||
The Hibari documentation has a brief photo illustration of how random
|
||||
slicing works, see [[http://hibari.github.io/hibari-doc/hibari-sysadmin-guide.en.html#chain-migration][Hibari Sysadmin Guide, chain migration]].
|
||||
|
||||
* 4. Use of the cluster namespace: name separation plus chain type
|
||||
|
||||
Let us assume that the cluster framework provides several different types
|
||||
of chains:
|
||||
|
||||
| Chain length | Namespace | Consistency Mode | Comment |
|
||||
|--------------+--------------+------------------+----------------------------------|
|
||||
| 3 | ~normal~ | eventual | Normal storage redundancy & cost |
|
||||
| 2 | ~reduced~ | eventual | Reduced cost storage |
|
||||
| 1 | ~risky~ | eventual | Really, really cheap storage |
|
||||
| 7 | ~paranoid~ | eventual | Safety-critical storage |
|
||||
| 3 | ~sequential~ | strong | Strong consistency |
|
||||
|--------------+--------------+------------------+----------------------------------|
|
||||
|
||||
The client may want to choose the amount of redundancy that its
|
||||
application requires: normal, reduced cost, or perhaps even a single
|
||||
copy. The cluster namespace is used by the client to signal this
|
||||
intention.
|
||||
|
||||
Further, the cluster administrators may wish to use the namespace to
|
||||
provide separate storage for different applications. Jane's
|
||||
application may use the namespace "jane-normal" and Bob's app uses
|
||||
"bob-reduced". Administrators may definine separate groups of
|
||||
chains on separate servers to serve these two applications.
|
||||
|
||||
* 5. In its lifetime, a file may be moved to different chains
|
||||
|
||||
The cluster management scheme may decide that files need to migrate to
|
||||
other chains -- i.e., file that is initially created on chain ID ~X~
|
||||
has been moved to chain ID ~Y~.
|
||||
|
||||
+ For storage load or I/O load balancing reasons.
|
||||
+ Because a chain is being decommissioned by the sysadmin.
|
||||
|
||||
* 6. Floating point is not required ... it is merely convenient for explanation
|
||||
|
||||
NOTE: Use of floating point terms is not required. For example,
|
||||
integer arithmetic could be used, if using a sufficiently large
|
||||
interval to create an even & smooth distribution of hashes across the
|
||||
expected maximum number of chains.
|
||||
|
||||
For example, if the maximum cluster size would be 4,000 individual
|
||||
Machi chains, then a minimum of 12 bits of integer space is required
|
||||
to assign one integer per Machi chain. However, for load balancing
|
||||
purposes, a finer grain of (for example) 100 integers per Machi
|
||||
chain would permit file migration to move increments of
|
||||
approximately 1% of single Machi chain's storage capacity. A
|
||||
minimum of 12+7=19 bits of hash space would be necessary to accommodate
|
||||
these constraints.
|
||||
|
||||
It is likely that Machi's final implementation will choose a 24 bit
|
||||
integer (or perhaps 32 bits) to represent the cluster locator.
|
||||
|
||||
* 7. Proposal: Break the opacity of Machi file names, slightly.
|
||||
|
||||
Machi assigns file names based on:
|
||||
|
||||
~ClientSuppliedPrefix ++ "^" ++ SomeOpaqueFileNameSuffix~
|
||||
|
||||
What if some parts of the system could peek inside of the opaque file name
|
||||
suffix in order to look at the cluster location information that we might
|
||||
code in the filename suffix?
|
||||
|
||||
We break the system into parts that speak two levels of protocols,
|
||||
"high" and "low".
|
||||
|
||||
+ The high level protocol is used outside of the Machi cluster
|
||||
+ The low level protocol is used inside of the Machi cluster
|
||||
|
||||
Both protocols are based on a Protocol Buffers specification and
|
||||
implementation. Other protocols, such as HTTP, will be added later.
|
||||
|
||||
#+BEGIN_SRC
|
||||
+-----------------------+
|
||||
| Machi external client |
|
||||
| e.g. Riak CS |
|
||||
+-----------------------+
|
||||
^
|
||||
| Machi "high" API
|
||||
| ProtoBuffs protocol Machi cluster boundary: outside
|
||||
.........................................................................
|
||||
| Machi cluster boundary: inside
|
||||
v
|
||||
+--------------------------+ +------------------------+
|
||||
| Machi "high" API service | | Machi HTTP API service |
|
||||
+--------------------------+ +------------------------+
|
||||
^ |
|
||||
| +------------------------+
|
||||
v v
|
||||
+------------------------+
|
||||
| Cluster bridge service |
|
||||
+------------------------+
|
||||
^
|
||||
| Machi "low" API
|
||||
| ProtoBuffs protocol
|
||||
+----------------------------------------+----+----+
|
||||
| | | |
|
||||
v v v v
|
||||
+-------------------------+ ... other chains...
|
||||
| Chain C1 (logical view) |
|
||||
| +--------------+ |
|
||||
| | FLU server 1 | |
|
||||
| | +--------------+ |
|
||||
| +--| FLU server 2 | |
|
||||
| +--------------+ | In reality, API bridge talks directly
|
||||
+-------------------------+ to each FLU server in a chain.
|
||||
#+END_SRC
|
||||
|
||||
** The notation we use
|
||||
|
||||
- ~N~ = the cluster namespace, chosen by the client.
|
||||
- ~p~ = file prefix, chosen by the client.
|
||||
- ~L~ = the cluster locator (a number, type is implementation-dependent)
|
||||
- ~Map~ = a mapping of cluster locators to chains
|
||||
- ~T~ = the target chain ID/name
|
||||
- ~u~ = a unique opaque file name suffix, e.g. a GUID string
|
||||
- ~F~ = a Machi file name, i.e., a concatenation of ~p^L^N^u~
|
||||
|
||||
** The details: cluster file append
|
||||
|
||||
0. Cluster client chooses ~N~ and ~p~ (i.e., cluster namespace and
|
||||
file prefix) and sends the append request to a Machi cluster member
|
||||
via the Protocol Buffers "high" API.
|
||||
1. Cluster bridge chooses ~T~ (i.e., target chain), based on criteria
|
||||
such as disk utilization percentage.
|
||||
2. Cluster bridge knows the cluster ~Map~ for namespace ~N~.
|
||||
3. Cluster bridge choose some cluster locator value ~L~ such that
|
||||
~rs_hash_with_float(L,Map) = T~ (see algorithm below).
|
||||
4. Cluster bridge sends its request to chain
|
||||
~T~: ~append_chunk(p,L,N,...) -> {ok,p^L^N^u,ByteOffset}~
|
||||
5. Cluster bridge forwards the reply tuple to the client.
|
||||
6. Client stores/uses the file name ~F = p^L^N^u~.
|
||||
|
||||
** The details: Cluster file read
|
||||
|
||||
0. Cluster client sends the read request to a Machi cluster member via
|
||||
the Protocol Buffers "high" API.
|
||||
1. Cluster bridge parses the file name ~F~ to find
|
||||
the values of ~L~ and ~N~ (recall, ~F = p^L^N^u~).
|
||||
2. Cluster bridge knows the Cluster ~Map~ for type ~N~.
|
||||
3. Cluster bridge calculates ~rs_hash_with_float(L,Map) = T~
|
||||
4. Cluster bridge sends request to chain ~T~:
|
||||
~read_chunk(F,...) ->~ ... reply
|
||||
5. Cluster bridge forwards the reply to the client.
|
||||
|
||||
** The details: calculating 'L' (the cluster locator number) to match a desired target chain
|
||||
|
||||
1. We know ~Map~, the current cluster mapping for a cluster namespace ~N~.
|
||||
2. We look inside of ~Map~, and we find all of the unit interval ranges
|
||||
that map to our desired target chain ~T~. Let's call this list
|
||||
~MapList = [Range1=(start,end],Range2=(start,end],...]~.
|
||||
3. In our example, ~T=Chain2~. The example ~Map~ contains a single
|
||||
unit interval range for ~Chain2~, ~[(0.33,0.58]]~.
|
||||
4. Choose a uniformly random number ~r~ on the unit interval.
|
||||
5. Calculate the cluster locator ~L~ by mapping ~r~ onto the concatenation
|
||||
of the cluster hash space range intervals in ~MapList~. For example,
|
||||
if ~r=0.5~, then ~L = 0.33 + 0.5*(0.58-0.33) = 0.455~, which is
|
||||
exactly in the middle of the ~(0.33,0.58]~ interval.
|
||||
|
||||
** A bit more about the cluster namespaces's meaning and use
|
||||
|
||||
For use by Riak CS, for example, we'd likely start with the following
|
||||
namespaces ... working our way down the list as we add new features
|
||||
and/or re-implement existing CS features.
|
||||
|
||||
- "standard" = Chain length = 3, eventually consistency mode
|
||||
- "reduced" = Chain length = 2, eventually consistency mode.
|
||||
- "stanchion7" = Chain length = 7, strong consistency mode. Perhaps
|
||||
use this namespace for the metadata required to re-implement the
|
||||
operations that are performed by today's Stanchion application.
|
||||
|
||||
We want the cluster framework to:
|
||||
|
||||
- provide means of creating and managing
|
||||
chains of different types, e.g., chain length, consistency mode.
|
||||
- manage the mapping of cluster namespace
|
||||
names to the chains in the system.
|
||||
- provide query functions to map a cluster
|
||||
namespace name to a cluster map,
|
||||
e.g. ~get_cluster_latest_map("reduced") -> Map{generation=7,...}~.
|
||||
|
||||
* 8. File migration (a.k.a. rebalancing/reparitioning/resharding/redistribution)
|
||||
|
||||
** What is "migration"?
|
||||
|
||||
This section describes Machi's file migration. Other storage systems
|
||||
call this process as "rebalancing", "repartitioning", "resharding" or
|
||||
"redistribution".
|
||||
For Riak Core applications, it is called "handoff" and "ring resizing"
|
||||
(depending on the context).
|
||||
See also the [[http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer][Hadoop file balancer]] for another example of a data
|
||||
migration process.
|
||||
|
||||
As discussed in section 5, the client can have good reason for wanting
|
||||
to have some control of the initial location of the file within the
|
||||
chain. However, the chain manager has an ongoing interest in
|
||||
balancing resources throughout the lifetime of the file. Disks will
|
||||
get full, hardware will change, read workload will fluctuate,
|
||||
etc etc.
|
||||
|
||||
This document uses the word "migration" to describe moving data from
|
||||
one Machi chain to another chain within a cluster system.
|
||||
|
||||
A simple variation of the Random Slicing hash algorithm can easily
|
||||
accommodate Machi's need to migrate files without interfering with
|
||||
availability. Machi's migration task is much simpler due to the
|
||||
immutable nature of Machi file data.
|
||||
|
||||
** Change to Random Slicing
|
||||
|
||||
The map used by the Random Slicing hash algorithm needs a few simple
|
||||
changes to make file migration straightforward.
|
||||
|
||||
- Add a "generation number", a strictly increasing number (similar to
|
||||
a Machi chain's "epoch number") that reflects the history of
|
||||
changes made to the Random Slicing map
|
||||
- Use a list of Random Slicing maps instead of a single map, one map
|
||||
per chance that files may not have been migrated yet out of
|
||||
that map.
|
||||
|
||||
As an example:
|
||||
|
||||
#+CAPTION: Illustration of 'Map', using four Machi chains
|
||||
|
||||
[[./migration-3to4.png]]
|
||||
|
||||
And the new Random Slicing map for some cluster namespace ~N~ might look
|
||||
like this:
|
||||
|
||||
| Generation number / Namespace | 7 / reduced |
|
||||
|-------------------------------+-------------|
|
||||
| SubMap | 1 |
|
||||
|-------------------------------+-------------|
|
||||
| Hash range | Chain ID |
|
||||
|-------------------------------+-------------|
|
||||
| 0.00 - 0.33 | Chain1 |
|
||||
| 0.33 - 0.66 | Chain2 |
|
||||
| 0.66 - 1.00 | Chain3 |
|
||||
|-------------------------------+-------------|
|
||||
| SubMap | 2 |
|
||||
|-------------------------------+-------------|
|
||||
| Hash range | Chain ID |
|
||||
|-------------------------------+-------------|
|
||||
| 0.00 - 0.25 | Chain1 |
|
||||
| 0.25 - 0.33 | Chain4 |
|
||||
| 0.33 - 0.58 | Chain2 |
|
||||
| 0.58 - 0.66 | Chain4 |
|
||||
| 0.66 - 0.91 | Chain3 |
|
||||
| 0.91 - 1.00 | Chain4 |
|
||||
|
||||
When a new Random Slicing map contains a single submap, then its use
|
||||
is identical to the original Random Slicing algorithm. If the map
|
||||
contains multiple submaps, then the access rules change a bit:
|
||||
|
||||
- Write operations always go to the newest/largest submap.
|
||||
- Read operations attempt to read from all unique submaps.
|
||||
- Skip searching submaps that refer to the same chain ID.
|
||||
- In this example, unit interval value 0.10 is mapped to Chain1
|
||||
by both submaps.
|
||||
- Read from newest/largest submap to oldest/smallest submap.
|
||||
- If not found in any submap, search a second time (to handle races
|
||||
with file copying between submaps).
|
||||
- If the requested data is found, optionally copy it directly to the
|
||||
newest submap. (This is a variation of read repair (RR). RR here
|
||||
accelerates the migration process and can reduce the number of
|
||||
operations required to query servers in multiple submaps).
|
||||
|
||||
The cluster manager is responsible for:
|
||||
|
||||
- Managing the various generations of the cluster Random Slicing maps for
|
||||
all namespaces.
|
||||
- Distributing namespace maps to cluster bridges.
|
||||
- Managing the processes that are responsible for copying "cold" data,
|
||||
i.e., files data that is not regularly accessed, to its new submap
|
||||
location.
|
||||
- When migration of a file to its new chain is confirmed successful,
|
||||
delete it from the old chain.
|
||||
|
||||
In example map #7, the cluster manager will copy files with unit interval
|
||||
assignments in ~(0.25,0.33]~, ~(0.58,0.66]~, and ~(0.91,1.00]~ from their
|
||||
old locations in chain IDs Chain1/2/3 to their new chain,
|
||||
Chain4. When the cluster manager is satisfied that all such files have
|
||||
been copied to Chain4, then the cluster manager can create and
|
||||
distribute a new map, such as:
|
||||
|
||||
| Generation number / Namespace | 8 / reduced |
|
||||
|-------------------------------+-------------|
|
||||
| SubMap | 1 |
|
||||
|-------------------------------+-------------|
|
||||
| Hash range | Chain ID |
|
||||
|-------------------------------+-------------|
|
||||
| 0.00 - 0.25 | Chain1 |
|
||||
| 0.25 - 0.33 | Chain4 |
|
||||
| 0.33 - 0.58 | Chain2 |
|
||||
| 0.58 - 0.66 | Chain4 |
|
||||
| 0.66 - 0.91 | Chain3 |
|
||||
| 0.91 - 1.00 | Chain4 |
|
||||
|
||||
The HibariDB system performs data migrations in almost exactly this
|
||||
manner. However, one important
|
||||
limitation of HibariDB is not being able to
|
||||
perform more than one migration at a time. HibariDB's data is
|
||||
mutable. Mutation causes many problems when migrating data
|
||||
across two submaps; three or more submaps was too complex to implement
|
||||
quickly and correctly.
|
||||
|
||||
Fortunately for Machi, its file data is immutable and therefore can
|
||||
easily manage many migrations in parallel, i.e., its submap list may
|
||||
be several maps long, each one for an in-progress file migration.
|
||||
|
||||
* 9. Other considerations for FLU/sequencer implementations
|
||||
|
||||
** Append to existing file when possible
|
||||
|
||||
The sequencer should always assign new offsets to the latest/newest
|
||||
file for any prefix, as long as all prerequisites are also true,
|
||||
|
||||
- The epoch has not changed. (In AP mode, epoch change -> mandatory
|
||||
file name suffix change.)
|
||||
- The cluster locator number is stable.
|
||||
- The latest file for prefix ~p~ is smaller than maximum file size for
|
||||
a FLU's configuration.
|
||||
|
||||
The stability of the cluster locator number is an implementation detail that
|
||||
must be managed by the cluster bridge.
|
||||
|
||||
Reuse of the same file is not possible if the bridge always chooses a
|
||||
different cluster locator number ~L~ or if the client always uses a unique
|
||||
file prefix ~p~. The latter is a sign of a misbehaved client; the
|
||||
former is a poorly-implemented bridge.
|
||||
|
||||
* 10. Acknowledgments
|
||||
|
||||
The original source for the "migration-4.png" and "migration-3to4.png" images
|
||||
come from the [[http://hibari.github.io/hibari-doc/images/migration-3to4.png][HibariDB documentation]].
|
||||
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
# Clone and compile Machi
|
||||
|
||||
Clone the Machi source repo and compile the source and test code. Run
|
||||
the following commands at your login shell:
|
||||
|
||||
cd /tmp
|
||||
git clone https://github.com/basho/machi.git
|
||||
cd machi
|
||||
git checkout master
|
||||
make # or 'gmake' if GNU make uses an alternate name
|
||||
|
||||
Then run the unit test suite. This may take up to two minutes or so
|
||||
to finish.
|
||||
|
||||
make test
|
||||
|
||||
At the end, the test suite should report that all tests passed. The
|
||||
actual number of tests shown in the "All `X` tests passed" line may be
|
||||
different than the example below.
|
||||
|
||||
[... many lines omitted ...]
|
||||
module 'event_logger'
|
||||
module 'chain_mgr_legacy'
|
||||
=======================================================
|
||||
All 90 tests passed.
|
||||
|
||||
If you had a test failure, a likely cause may be a limit on the number
|
||||
of file descriptors available to your user process. (Recent releases
|
||||
of OS X have a limit of 1024 file descriptors, which may be too slow.)
|
||||
The output of the `limit -n` will tell you your file descriptor limit.
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
## Machi developer environment prerequisites
|
||||
|
||||
1. Machi requires an 64-bit variant of UNIX: OS X, FreeBSD, Linux, or
|
||||
Solaris machine is a standard developer environment for C and C++
|
||||
applications (64-bit versions).
|
||||
2. You'll need the `git` source management utility.
|
||||
3. You'll need the 64-bit Erlang/OTP 17 runtime environment. Please
|
||||
don't use earlier or later versions until we have a chance to fix
|
||||
the compilation warnings that versions R16B and 18 will trigger.
|
||||
Also, please verify that you are not using a 32-bit Erlang/OTP
|
||||
runtime package.
|
||||
|
||||
For `git` and the Erlang runtime, please use your OS-specific
|
||||
package manager to install these. If your package manager doesn't
|
||||
have 64-bit Erlang/OTP version 17 available, then we recommend using the
|
||||
[precompiled packages available at Erlang Solutions](https://www.erlang-solutions.com/resources/download.html).
|
||||
|
||||
Also, please verify that you have enough file descriptors available to
|
||||
your user processes. The output of `ulimit -n` should report at least
|
||||
4,000 file descriptors available. If your limit is lower (a frequent
|
||||
problem for OS X users), please increase it to at least 4,000.
|
||||
|
||||
# Using Vagrant to set up a developer environment for Machi
|
||||
|
||||
The Machi source directory contains a `Vagrantfile` for creating an
|
||||
Ubuntu Linux-based virtual machine for compiling and running Machi.
|
||||
This file is in the
|
||||
[$SRC_TOP/priv/humming-consensus-demo.vagrant](../priv/humming-consensus-demo.vagrant)
|
||||
directory.
|
||||
|
||||
If used as-is, the virtual machine specification is modest.
|
||||
|
||||
* 1 virtual CPU
|
||||
* 512MB virtual memory
|
||||
* 768MB swap space
|
||||
* 79GB sparse virtual disk image. After installing prerequisites and
|
||||
compiling Machi, the root file system uses approximately 2.7 GBytes.
|
||||
|
||||
|
|
@ -1,617 +0,0 @@
|
|||
FLU and Chain Life Cycle Management -*- mode: org; -*-
|
||||
#+STARTUP: lognotedone hidestars indent showall inlineimages
|
||||
#+COMMENT: To generate the outline section: egrep '^\*[*]* ' doc/flu-and-chain-lifecycle.org | egrep -v '^\* Outline' | sed -e 's/^\*\*\* / + /' -e 's/^\*\* / + /' -e 's/^\* /+ /'
|
||||
|
||||
* FLU and Chain Life Cycle Management
|
||||
|
||||
In an ideal world, we (the Machi development team) would have a full
|
||||
vision of how Machi would be managed, down to the last detail of
|
||||
beautiful CLI character and network protocol bit. Our vision isn't
|
||||
complete yet, so we are working one small step at a time.
|
||||
|
||||
* Outline
|
||||
|
||||
+ FLU and Chain Life Cycle Management
|
||||
+ Terminology review
|
||||
+ Terminology: Machi run-time components/services/thingies
|
||||
+ Terminology: Machi chain data structures
|
||||
+ Terminology: Machi cluster data structures
|
||||
+ Overview of administrative life cycles
|
||||
+ Cluster administrative life cycle
|
||||
+ Chain administrative life cycle
|
||||
+ FLU server administrative life cycle
|
||||
+ Quick admin: declarative management of Machi FLU and chain life cycles
|
||||
+ Quick admin uses the "rc.d" config scheme for life cycle management
|
||||
+ Quick admin's declarative "language": an Erlang-flavored AST
|
||||
+ Term 'host': define a new host for FLU services
|
||||
+ Term 'flu': define a new FLU
|
||||
+ Term 'chain': define or reconfigure a chain
|
||||
+ Executing quick admin AST files via the 'machi-admin' utility
|
||||
+ Checking the syntax of an AST file
|
||||
+ Executing an AST file
|
||||
+ Using quick admin to manage multiple machines
|
||||
+ The "rc.d" style configuration file scheme
|
||||
+ Riak had a similar configuration file editing problem (and its solution)
|
||||
+ Machi's "rc.d" file scheme.
|
||||
+ FLU life cycle management using "rc.d" style files
|
||||
+ The key configuration components of a FLU
|
||||
+ Chain life cycle management using "rc.d" style files
|
||||
+ The key configuration components of a chain
|
||||
|
||||
* Terminology review
|
||||
|
||||
** Terminology: Machi run-time components/services/thingies
|
||||
|
||||
+ FLU: a basic Machi server, responsible for managing a collection of
|
||||
files.
|
||||
|
||||
+ Chain: a small collection of FLUs that maintain replicas of the same
|
||||
collection of files. A chain is usually small, 1-3 servers, where
|
||||
more than 3 would be used only in cases when availability of
|
||||
certain data is critical despite failures of several machines.
|
||||
+ The length of a chain is directly proportional to its
|
||||
replication factor, e.g., a chain length=3 will maintain
|
||||
(nominally) 3 replicas of each file.
|
||||
+ To maintain file availability when ~F~ failures have occurred, a
|
||||
chain must be at least ~F+1~ members long. (In comparison, the
|
||||
quorum replication technique requires ~2F+1~ members in the
|
||||
general case.)
|
||||
|
||||
+ Cluster: A collection of Machi chains that are used to store files
|
||||
in a horizontally partitioned/sharded/distributed manner.
|
||||
|
||||
** Terminology: Machi data structures
|
||||
|
||||
+ Projection: used to define a single chain: the chain's consistency
|
||||
mode (strong or eventual consistency), all members (from an
|
||||
administrative point of view), all active members (from a runtime,
|
||||
automatically-managed point of view), repairing/file-syncing members
|
||||
(also runtime, auto-managed), and so on
|
||||
|
||||
+ Epoch: A version number of a projection. The epoch number is used
|
||||
by both clients & servers to manage transitions from one projection
|
||||
to another, e.g., when the chain is temporarily shortened by the
|
||||
failure of a member FLU server.
|
||||
|
||||
** Terminology: Machi cluster data structures
|
||||
|
||||
+ Namespace: A collection of human-friendly names that are mapped to
|
||||
groups of Machi chains that provide the same type of storage
|
||||
service: consistency mode, replication policy, etc.
|
||||
+ A single namespace name, e.g. ~normal-ec~, is paired with a single
|
||||
cluster map (see below).
|
||||
+ Example: ~normal-ec~ might be a collection of Machi chains in
|
||||
eventually-consistent mode that are of length=3.
|
||||
+ Example: ~risky-ec~ might be a collection of Machi chains in
|
||||
eventually-consistent mode that are of length=1.
|
||||
+ Example: ~mgmt-critical~ might be a collection of Machi chains in
|
||||
strongly-consistent mode that are of length=7.
|
||||
|
||||
+ Cluster map: Encodes the rules which partition/shard/distribute
|
||||
the files stored in a particular namespace across a group of chains
|
||||
that collectively store the namespace's files.
|
||||
|
||||
+ Chain weight: A value assigned to each chain within a cluster map
|
||||
structure that defines the relative storage capacity of a chain
|
||||
within the namespace. For example, a chain weight=150 has 50% more
|
||||
capacity than a chain weight=100.
|
||||
|
||||
+ Cluster map epoch: The version number assigned to a cluster map.
|
||||
|
||||
* Overview of administrative life cycles
|
||||
|
||||
** Cluster administrative life cycle
|
||||
|
||||
+ Cluster is first created
|
||||
+ Adds namespaces (e.g. consistency policy + chain length policy) to
|
||||
the cluster
|
||||
+ Chains are added to/removed from a namespace to increase/decrease the
|
||||
namespace's storage capacity.
|
||||
+ Adjust chain weights within a namespace, e.g., to shift files
|
||||
within the namespace to chains with greater storage capacity
|
||||
resources and/or runtime I/O resources.
|
||||
|
||||
A cluster "file migration" is the process of moving files from one
|
||||
namespace member chain to another for purposes of shifting &
|
||||
re-balancing storage capacity and/or runtime I/O capacity.
|
||||
|
||||
** Chain administrative life cycle
|
||||
|
||||
+ A chain is created with an initial FLU membership list.
|
||||
+ Chain may be administratively modified zero or more times to
|
||||
add/remove member FLU servers.
|
||||
+ A chain may be decommissioned.
|
||||
|
||||
See also: http://basho.github.io/machi/edoc/machi_lifecycle_mgr.html
|
||||
|
||||
** FLU server administrative life cycle
|
||||
|
||||
+ A FLU is created after an administrator chooses the FLU's runtime
|
||||
location is selected by the administrator: which machine/virtual
|
||||
machine, IP address and TCP port allocation, etc.
|
||||
+ An unassigned FLU may be added to a chain by chain administrative
|
||||
policy.
|
||||
+ A FLU that is assigned to a chain may be removed from that chain by
|
||||
chain administrative policy.
|
||||
+ In the current implementation, the FLU's Erlang processes will be
|
||||
halted. Then the FLU's data and metadata files will be moved to
|
||||
another area of the disk for safekeeping. Later, a "garbage
|
||||
collection" process can be used for reclaiming disk space used by
|
||||
halted FLU servers.
|
||||
|
||||
See also: http://basho.github.io/machi/edoc/machi_lifecycle_mgr.html
|
||||
|
||||
* Quick admin: declarative management of Machi FLU and chain life cycles
|
||||
|
||||
The "quick admin" scheme is a temporary (?) tool for managing Machi
|
||||
FLU server and chain life cycles in a declarative manner. The API is
|
||||
described in this section.
|
||||
|
||||
** Quick admin uses the "rc.d" config scheme for life cycle management
|
||||
|
||||
As described at the top of
|
||||
http://basho.github.io/machi/edoc/machi_lifecycle_mgr.html, the "rc.d"
|
||||
config files do not manage "policy". "Policy" is doing the right
|
||||
thing with a Machi cluster from a systems administrator's
|
||||
point of view. The "rc.d" config files can only implement decisions
|
||||
made according to policy.
|
||||
|
||||
The "quick admin" tool is a first attempt at automating policy
|
||||
decisions in a safe way (we hope) that is also easy to implement (we
|
||||
hope) with a variety of systems management tools, e.g. Chef, Puppet,
|
||||
Ansible, Saltstack, or plain-old-human-at-a-keyboard.
|
||||
|
||||
** Quick admin's declarative "language": an Erlang-flavored AST
|
||||
|
||||
The "language" that an administrator uses to express desired policy
|
||||
changes is not (yet) a true language. As a quick implementation hack,
|
||||
the current language is an Erlang-flavored abstract syntax tree
|
||||
(AST). The tree isn't very deep, either, frequently just one
|
||||
element tall. (Not much of a tree, is it?)
|
||||
|
||||
There are three terms in the language currently:
|
||||
|
||||
+ ~host~, define a new host that can execute FLU servers
|
||||
+ ~flu~, define a new FLU
|
||||
+ ~chain~, define a new chain or re-configure an existing chain with
|
||||
the same name
|
||||
|
||||
*** Term 'host': define a new host for FLU services
|
||||
|
||||
In this context, a host is a machine, virtual machine, or container
|
||||
that can execute the Machi application and can therefore provide FLU
|
||||
services, i.e. file service, Humming Consensus management.
|
||||
|
||||
Two formats may be used to define a new host:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{host, Name, Props}.
|
||||
{host, Name, AdminI, ClientI, Props}.
|
||||
#+END_SRC
|
||||
|
||||
The shorter tuple is shorthand notation for the latter. If the
|
||||
shorthand form is used, then it will be converted automatically to the
|
||||
long form as:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{host, Name, AdminI=Name, ClientI=Name, Props}.
|
||||
#+END_SRC
|
||||
|
||||
Type information, description, and restrictions:
|
||||
|
||||
+ ~Name::string()~ The ~Name~ attribute must be unique. Note that it
|
||||
is possible to define two different hosts, one using a DNS hostname
|
||||
and one using an IP address. The user must avoid this
|
||||
double-definition because it is not enforced by quick admin.
|
||||
+ The ~Name~ field is used for cross-reference purposes with other
|
||||
terms, e.g., ~flu~ and ~chain~.
|
||||
+ There is no syntax yet for removing a host definition.
|
||||
|
||||
+ ~AdminI::string()~ A DNS hostname or IP address for cluster
|
||||
administration purposes, e.g. SSH access.
|
||||
+ This field is unused at the present time.
|
||||
|
||||
+ ~ClientI::string()~ A DNS hostname or IP address for Machi's client
|
||||
protocol access, e.g., Protocol Buffers network API service.
|
||||
+ This field is unused at the present time.
|
||||
|
||||
+ ~props::proplist()~ is an Erlang-style property list for specifying
|
||||
additional configuration options, debugging information, sysadmin
|
||||
comments, etc.
|
||||
|
||||
+ A full-featured admin tool should also include managing several
|
||||
other aspects of configuration related to a "host". For example,
|
||||
for any single IP address, quick admin assumes that there will be
|
||||
exactly one Erlang VM that is running the Machi application. Of
|
||||
course, it is possible to have dozens of Erlang VMs on the same
|
||||
(let's assume for clarity) hardware machine and all running Machi
|
||||
... but there are additional aspects of such a machine that quick
|
||||
admin does not account for
|
||||
+ multiple IP addresses per machine
|
||||
+ multiple Machi package installation paths
|
||||
+ multiple Machi config files (e.g. cuttlefish config, ~etc.conf~,
|
||||
~vm.args~)
|
||||
+ multiple data directories/file system mount points
|
||||
+ This is also a management problem for quick admin for a single
|
||||
Machi package on a machine to take advantage of bulk data
|
||||
storage using multiple multiple file system mount points.
|
||||
+ multiple Erlang VM host names, required for distributed Erlang,
|
||||
which is used for communication with ~machi~ and ~machi-admin~
|
||||
command line utilities.
|
||||
+ and others....
|
||||
|
||||
*** Term 'flu': define a new FLU
|
||||
|
||||
A new FLU is defined relative to a previously-defined ~host~ entities;
|
||||
an exception will be thrown if the ~host~ cannot be cross-referenced.
|
||||
|
||||
#+BEGIN_SRC
|
||||
{flu, Name, HostName, Port, Props}
|
||||
#+END_SRC
|
||||
|
||||
Type information, description, and restrictions:
|
||||
|
||||
+ ~Name::atom()~ The name of the FLU, as a human-friendly name and
|
||||
also for internal management use; please note the ~atom()~ type.
|
||||
This name must be unique.
|
||||
+ The ~Name~ field is used for cross-reference purposes with the
|
||||
~chain~ term.
|
||||
+ There is no syntax yet for removing a FLU definition.
|
||||
|
||||
+ ~Hostname::string()~ The cross-reference name of the ~host~ that
|
||||
this FLU should run on.
|
||||
|
||||
+ ~Port::non_neg_integer()~ The TCP port used by this FLU server's
|
||||
Protocol Buffers network API listener service
|
||||
|
||||
+ ~props::proplist()~ is an Erlang-style property list for specifying
|
||||
additional configuration options, debugging information, sysadmin
|
||||
comments, etc.
|
||||
|
||||
*** Term 'chain': define or reconfigure a chain
|
||||
|
||||
A chain is defined relative to zero or more previously-defined ~flu~
|
||||
entities; an exception will be thrown if any ~flu~ cannot be
|
||||
cross-referenced.
|
||||
|
||||
Two formats may be used to define/reconfigure a chain:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{chain, Name, FullList, Props}.
|
||||
{chain, Name, CMode, FullList, Witnesses, Props}.
|
||||
#+END_SRC
|
||||
|
||||
The shorter tuple is shorthand notation for the latter. If the
|
||||
shorthand form is used, then it will be converted automatically to the
|
||||
long form as:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{chain, Name, ap_mode, FullList, [], Props}.
|
||||
#+END_SRC
|
||||
|
||||
Type information, description, and restrictions:
|
||||
|
||||
+ ~Name::atom()~ The name of the chain, as a human-friendly name and
|
||||
also for internal management use; please note the ~atom()~ type.
|
||||
This name must be unique.
|
||||
+ There is no syntax yet for removing a chain definition.
|
||||
|
||||
+ ~CMode::'ap_mode'|'cp_mode'~ Defines the consistency mode of the
|
||||
chain, either eventual consistency or strong consistency,
|
||||
respectively.
|
||||
+ A chain cannot change consistency mode, e.g., from
|
||||
strong~->~eventual consistency.
|
||||
|
||||
+ ~FullList::list(atom())~ Specifies the list of full-service FLU
|
||||
servers, i.e. servers that provide file data & metadata services as
|
||||
well as Humming Consensus. Each atom in the list must
|
||||
cross-reference with a previously defined ~chain~; an exception will
|
||||
be thrown if any ~flu~ cannot be cross-referenced.
|
||||
|
||||
+ ~Witnesses::list(atom())~ Specifies the list of witness-only
|
||||
servers, i.e. servers that only participate in Humming Consensus.
|
||||
Each atom in the list must cross-reference with a previously defined
|
||||
~chain~; an exception will be thrown if any ~flu~ cannot be
|
||||
cross-referenced.
|
||||
+ This list must be empty for eventual consistency chains.
|
||||
|
||||
+ ~props::proplist()~ is an Erlang-style property list for specifying
|
||||
additional configuration options, debugging information, sysadmin
|
||||
comments, etc.
|
||||
|
||||
+ If this term specifies a new ~chain~ name, then all of the member
|
||||
FLU servers (full & witness types) will be bootstrapped to a
|
||||
starting configuration.
|
||||
|
||||
+ If this term specifies a previously-defined ~chain~ name, then all
|
||||
of the member FLU servers (full & witness types, respectively) will
|
||||
be adjusted to add or remove members, as appropriate.
|
||||
+ Any FLU servers added to either list must not be assigned to any
|
||||
other chain, or they must be a member of this specific chain.
|
||||
+ Any FLU servers removed from either list will be halted.
|
||||
(See the "FLU server administrative life cycle" section above.)
|
||||
|
||||
** Executing quick admin AST files via the 'machi-admin' utility
|
||||
|
||||
Examples of quick admin AST files can be found in the
|
||||
~priv/quick-admin/examples~ directory. Below is an example that will
|
||||
define a new host ( ~"localhost"~ ), three new FLU servers ( ~f1~ & ~f2~
|
||||
and ~f3~ ), and an eventually consistent chain ( ~c1~ ) that uses the new
|
||||
FLU servers:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{host, "localhost", []}.
|
||||
{flu,f1,"localhost",20401,[]}.
|
||||
{flu,f2,"localhost",20402,[]}.
|
||||
{flu,f3,"localhost",20403,[]}.
|
||||
{chain,c1,[f1,f2,f3],[]}.
|
||||
#+END_SRC
|
||||
|
||||
*** Checking the syntax of an AST file
|
||||
|
||||
Given an AST config file, ~/path/to/ast/file~, its basic syntax and
|
||||
correctness can be checked without executing it.
|
||||
|
||||
#+BEGIN_SRC
|
||||
./rel/machi/bin/machi-admin quick-admin-check /path/to/ast/file
|
||||
#+END_SRC
|
||||
|
||||
+ The utility will exit with status zero and output ~ok~ if the syntax
|
||||
and proposed configuration appears to be correct.
|
||||
+ If there is an error, the utility will exit with status one, and an
|
||||
error message will be printed.
|
||||
|
||||
*** Executing an AST file
|
||||
|
||||
Given an AST config file, ~/path/to/ast/file~, it can be executed
|
||||
using the command:
|
||||
|
||||
#+BEGIN_SRC
|
||||
./rel/machi/bin/machi-admin quick-admin-apply /path/to/ast/file RelativeHost
|
||||
#+END_SRC
|
||||
|
||||
... where the last argument, ~RelativeHost~, should be the exact
|
||||
spelling of one of the previously defined AST ~host~ entities,
|
||||
*and also* is the same host that the ~machi-admin~ utility is being
|
||||
executed on.
|
||||
|
||||
Restrictions and warnings:
|
||||
|
||||
+ This is alpha quality software.
|
||||
|
||||
+ There is no "undo".
|
||||
+ Of course there is, but you need to resort to doing things like
|
||||
using ~machi attach~ to attach to the server's CLI to then execute
|
||||
magic Erlang incantations to stop FLUs, unconfigure chains, etc.
|
||||
+ Oh, and delete some files with magic paths, also.
|
||||
|
||||
** Using quick admin to manage multiple machines
|
||||
|
||||
A quick sketch follows:
|
||||
|
||||
1. Create the AST file to specify all of the changes that you wish to
|
||||
make to all hosts, FLUs, and/or chains, e.g., ~/tmp/ast.txt~.
|
||||
2. Check the basic syntax with the ~quick-admin-check~ argument to
|
||||
~machi-admin~.
|
||||
3. If the syntax is good, then copy ~/tmp/ast.txt~ to all hosts in the
|
||||
cluster, using the same path, ~/tmp/ast.txt~.
|
||||
4. For each machine in the cluster, run:
|
||||
#+BEGIN_SRC
|
||||
./rel/machi/bin/machi-admin quick-admin-apply /tmp/ast.txt RelativeHost
|
||||
#+END_SRC
|
||||
|
||||
... where RelativeHost is the AST ~host~ name of the machine that you
|
||||
are executing the ~machi-admin~ command on. The command should be
|
||||
successful, with exit status 0 and outputting the string ~ok~.
|
||||
|
||||
Finally, for each machine in the cluster, a listing of all files in
|
||||
the directory ~rel/machi/etc/quick-admin-archive~ should show exactly
|
||||
the same files, one for each time that ~quick-admin-apply~ has been
|
||||
run successfully on that machine.
|
||||
|
||||
* The "rc.d" style configuration file scheme
|
||||
|
||||
This configuration scheme is inspired by BSD UNIX's ~init(8)~ process
|
||||
manager's configuration style, called "rc.d" after the name of the
|
||||
directory where these files are stored, ~/etc/rc.d~. The ~init~
|
||||
process is responsible for (among other things) starting UNIX
|
||||
processes at machine boot time and stopping them when the machine is
|
||||
shut down.
|
||||
|
||||
The original scheme used by ~init~ to start processes at boot time was
|
||||
a single Bourne shell script called ~/etc/rc~. When a new software
|
||||
package was installed that required a daemon to be started at boot
|
||||
time, text was added to the ~/etc/rc~ file. Uninstalling packages was
|
||||
much trickier, because it meant removing lines from a file that
|
||||
*is a computer program (run by the Bourne shell, a Turing-complete
|
||||
programming language)*. Error-free editing of the ~/etc/rc~ script
|
||||
was impossible in all cases.
|
||||
|
||||
Later, ~init~'s configuration was split into a few master Bourne shell
|
||||
scripts and a subdirectory, ~/etc/rc.d~. The subdirectory contained
|
||||
shell scripts that were responsible for boot time starting of a single
|
||||
daemon or service, e.g. NFS or an HTTP server. When a new software
|
||||
package was added, a new file was added to the ~rc.d~ subdirectory.
|
||||
When a package was removed, the corresponding file in ~rc.d~ was
|
||||
removed. With this simple scheme, addition & removal of boot time
|
||||
scripts was vastly simplified.
|
||||
|
||||
** Riak had a similar configuration file editing problem (and its solution)
|
||||
|
||||
Another software product from Basho Technologies, Riak, had a similar
|
||||
configuration file editing problem. One file in particular,
|
||||
~app.config~, had a syntax that made it difficult both for human
|
||||
systems administrators and also computer programs to edit the file in
|
||||
a syntactically correct manner.
|
||||
|
||||
Later releases of Riak switched to an alternative configuration file
|
||||
format, one inspired by the BSD UNIX ~sysctl(8)~ utility and
|
||||
~sysctl.conf(5)~ file syntax. The ~sysctl.conf~ format is much easier
|
||||
to manage by computer programs to add items. Removing items is not
|
||||
100% simple, however: the correct lines must be identified and then
|
||||
removed (e.g. with Perl or a text editor or combination of ~grep -v~
|
||||
and ~mv~), but removing any comment lines that "belong" to the removed
|
||||
config item(s) is not any easy for a 1-line shell script to do 100%
|
||||
correctly.
|
||||
|
||||
Machi will use the ~sysctl.conf~ style configuration for some
|
||||
application configuration variables. However, adding & removing FLUs
|
||||
and chains will be managed using the "rc.d" style because of the
|
||||
"rc.d" scheme's simplicity and tolerance of mistakes by administrators
|
||||
(human or computer).
|
||||
|
||||
** Machi's "rc.d" file scheme.
|
||||
|
||||
Machi will use a single subdirectory that will contain configuration
|
||||
files for some life cycle management task, e.g. a single FLU or a
|
||||
single chain.
|
||||
|
||||
The contents of the file should be a single Erlang term, serialized in
|
||||
ASCII form as Erlang source code statement, i.e. a single Erlang term
|
||||
~T~ that is formatted by ~io:format("~w.",[T]).~. This file must be
|
||||
parseable by the Erlang function ~file:consult()~.
|
||||
|
||||
Later versions of Machi may change the file format to be more familiar
|
||||
to administrators who are unaccustomed to Erlang language syntax.
|
||||
|
||||
** FLU life cycle management using "rc.d" style files
|
||||
|
||||
*** The key configuration components of a FLU
|
||||
|
||||
1. The machine (or virtual machine) to run it on.
|
||||
2. The Machi software package's artifacts to execute.
|
||||
3. The disk device(s) used to store Machi file data & metadata, "rc.d"
|
||||
style config files, etc.
|
||||
4. The name, IP address and TCP port assigned to the FLU service.
|
||||
5. Its chain assignment.
|
||||
|
||||
Notes:
|
||||
|
||||
+ Items 1-3 are currently outside of the scope of this life cycle
|
||||
document. We assume that human administrators know how to do these
|
||||
things.
|
||||
+ Item 4's properties are explicitly managed by a FLU-defining "rc.d"
|
||||
style config file.
|
||||
+ Item 5 is managed by the chain life cycle management system.
|
||||
|
||||
Here is an example of a properly formatted FLU config file:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{p_srvr,f1,machi_flu1_client,"192.168.72.23",20401,[]}.
|
||||
#+END_SRC
|
||||
|
||||
... which corresponds to the following Erlang record definition:
|
||||
|
||||
#+BEGIN_SRC
|
||||
-record(p_srvr, {
|
||||
name :: atom(),
|
||||
proto_mod = 'machi_flu1_client' :: atom(), % Module name
|
||||
address :: term(), % Protocol-specific
|
||||
port :: term(), % Protocol-specific
|
||||
props = [] :: list() % proplist for other related info
|
||||
}).
|
||||
#+END_SRC
|
||||
|
||||
+ ~name~ is ~f1~. This is name of the FLU. This name should be
|
||||
unique over the lifetime of the administrative domain and thus
|
||||
managed by external policy. This name must be the same as the name
|
||||
of the config file that defines the FLU.
|
||||
+ ~proto_mod~ is used for internal management purposes and should be
|
||||
considered a mandatory constant.
|
||||
+ ~address~ is "192.168.72.23". The DNS hostname or IP address used
|
||||
by other servers to communicate with this FLU. This must be a valid
|
||||
IP address, previously assigned to this machine/VM using the
|
||||
appropriate operating system-specific procedure.
|
||||
+ ~port~ is TCP port 20401. The TCP port number that the FLU listens
|
||||
to for incoming Protocol Buffers-serialized communication. This TCP
|
||||
port must not be in use (now or in the future) by another Machi FLU
|
||||
or any other process running on this machine/VM.
|
||||
+ ~props~ is an Erlang-style property list for specifying additional
|
||||
configuration options, debugging information, sysadmin comments,
|
||||
etc.
|
||||
|
||||
** Chain life cycle management using "rc.d" style files
|
||||
|
||||
Unlike FLUs, chains have a self-management aspect that makes a chain
|
||||
life cycle different from a single FLU server. Machi's chains are
|
||||
self-managing, via Humming Consensus; see the
|
||||
https://github.com/basho/machi/tree/master/doc/ directory for much
|
||||
more detail about Humming Consensus. After FLUs have received their
|
||||
initial chain configuration for Humming Consensus, the FLUs will
|
||||
manage the chain (and each other) by themselves.
|
||||
|
||||
However, Humming Consensus does not handle three chain management
|
||||
problems:
|
||||
|
||||
1. Specifying the very first chain configuration,
|
||||
2. Altering the membership of the chain (i.e. adding/removing FLUs
|
||||
from the chain),
|
||||
3. Stopping the chain permanently.
|
||||
|
||||
A chain "rc.d" file will only be used to bootstrap a newly-defined FLU
|
||||
server. It's like a piece of glue information to introduce the new
|
||||
FLU to the Humming Consensus group that is managing the chain's
|
||||
dynamic state (e.g. which members are up or down). In all other
|
||||
respects, chain config files are ignored by life cycle management code.
|
||||
However, to mimic the life cycle of the FLU server's "rc.d" config
|
||||
files, a chain "rc.d" files is not deleted until the chain has been
|
||||
decommissioned (i.e. defined with length=0).
|
||||
|
||||
*** The key configuration components of a chain
|
||||
|
||||
1. The name of the chain.
|
||||
2. Consistency mode: eventually consistent or strongly consistent.
|
||||
3. The membership list of all FLU servers in the chain.
|
||||
+ Remember, all servers in a single chain will manage full replicas
|
||||
of the same collection of Machi files.
|
||||
4. If the chain is defined to use strongly consistent mode, then a
|
||||
list of "witness servers" may also be defined. See the
|
||||
[https://github.com/basho/machi/tree/master/doc/] documentation for
|
||||
more information on witness servers.
|
||||
+ The witness list must be empty for all chains in eventual
|
||||
consistency mode.
|
||||
|
||||
Here is an example of a properly formatted chain config file:
|
||||
|
||||
#+BEGIN_SRC
|
||||
{chain_def_v1,c1,ap_mode,
|
||||
[{p_srvr,f1,machi_flu1_client,"localhost",20401,[]},
|
||||
{p_srvr,f2,machi_flu1_client,"localhost",20402,[]},
|
||||
{p_srvr,f3,machi_flu1_client,"localhost",20403,[]}],
|
||||
[],[],[],
|
||||
[f1,f2,f3],
|
||||
[],[]}.
|
||||
#+END_SRC
|
||||
|
||||
... which corresponds to the following Erlang record definition:
|
||||
|
||||
#+BEGIN_SRC
|
||||
-record(chain_def_v1, {
|
||||
name :: atom(), % chain name
|
||||
mode :: 'ap_mode' | 'cp_mode',
|
||||
full = [] :: [p_srvr()],
|
||||
witnesses = [] :: [p_srvr()],
|
||||
old_full = [] :: [atom()], % guard against some races
|
||||
old_witnesses=[] :: [atom()], % guard against some races
|
||||
local_run = [] :: [atom()], % must be tailored to each machine!
|
||||
local_stop = [] :: [atom()], % must be tailored to each machine!
|
||||
props = [] :: list() % proplist for other related info
|
||||
}).
|
||||
#+END_SRC
|
||||
|
||||
+ ~name~ is ~c1~, the name of the chain. This name should be unique
|
||||
over the lifetime of the administrative domain and thus managed by
|
||||
external policy. This name must be the same as the name of the
|
||||
config file that defines the chain.
|
||||
+ ~mode~ is ~ap_mode~, an internal code symbol for eventual
|
||||
consistency mode.
|
||||
+ ~full~ is a list of Erlang ~#p_srvr{}~ records for full-service
|
||||
members of the chain, i.e., providing Machi file data & metadata
|
||||
storage services.
|
||||
+ ~witnesses~ is a list of Erlang ~#p_srvr{}~ records for witness-only
|
||||
FLU servers, i.e., providing only Humming Consensus service.
|
||||
+ The next four fields are used for internal management only.
|
||||
+ ~props~ is an Erlang-style property list for specifying additional
|
||||
configuration options, debugging information, sysadmin comments,
|
||||
etc.
|
||||
|
||||
Binary file not shown.
Binary file not shown.
|
|
@ -1,372 +0,0 @@
|
|||
|
||||
# Table of contents
|
||||
|
||||
* [Hands-on experiments with Machi and Humming Consensus](#hands-on)
|
||||
* [Using the network partition simulator and convergence demo test code](#partition-simulator)
|
||||
|
||||
<a name="hands-on">
|
||||
# Hands-on experiments with Machi and Humming Consensus
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Please refer to the
|
||||
[Machi development environment prerequisites doc](./dev-prerequisites.md)
|
||||
for Machi developer environment prerequisites.
|
||||
|
||||
If you do not have an Erlang/OTP runtime system available, but you do
|
||||
have [the Vagrant virtual machine](https://www.vagrantup.com/) manager
|
||||
available, then please refer to the instructions in the prerequisites
|
||||
doc for using Vagrant.
|
||||
|
||||
<a name="clone-compile">
|
||||
## Clone and compile the code
|
||||
|
||||
Please see the
|
||||
[Machi 'clone and compile' doc](./dev-clone-compile.md)
|
||||
for the short list of steps required to fetch the Machi source code
|
||||
from GitHub and to compile & test Machi.
|
||||
|
||||
## Running three Machi instances on a single machine
|
||||
|
||||
All of the commands that should be run at your login shell (e.g. Bash,
|
||||
c-shell) can be cut-and-pasted from this document directly to your
|
||||
login shell prompt.
|
||||
|
||||
Run the following command:
|
||||
|
||||
make stagedevrel
|
||||
|
||||
This will create a directory structure like this:
|
||||
|
||||
|-dev1-|... stand-alone Machi app + subdirectories
|
||||
|-dev-|-dev2-|... stand-alone Machi app + directories
|
||||
|-dev3-|... stand-alone Machi app + directories
|
||||
|
||||
Each of the `dev/dev1`, `dev/dev2`, and `dev/dev3` are stand-alone
|
||||
application instances of Machi and can be run independently of each
|
||||
other on the same machine. This demo will use all three.
|
||||
|
||||
The lifecycle management utilities for Machi are a bit immature,
|
||||
currently. They assume that each Machi server runs on a host with a
|
||||
unique hostname -- there is no flexibility built-in yet to easily run
|
||||
multiple Machi instances on the same machine. To continue with the
|
||||
demo, we need to use `sudo` or `su` to obtain superuser privileges to
|
||||
edit the `/etc/hosts` file.
|
||||
|
||||
Please add the following line to `/etc/hosts`, using this command:
|
||||
|
||||
sudo sh -c 'echo "127.0.0.1 machi1 machi2 machi3" >> /etc/hosts'
|
||||
|
||||
Next, we will use a shell script to finish setting up our cluster. It
|
||||
will do the following for us:
|
||||
|
||||
* Verify that the new line that was added to `/etc/hosts` is correct.
|
||||
* Modify the `etc/app.config` files to configure the Humming Consensus
|
||||
chain manager's actions logged to the `log/console.log` file.
|
||||
* Start the three application instances.
|
||||
* Verify that the three instances are running correctly.
|
||||
* Configure a single chain, with one FLU server per application
|
||||
instance.
|
||||
|
||||
Please run this script using this command:
|
||||
|
||||
./priv/humming-consensus-demo.setup.sh
|
||||
|
||||
If the output looks like this (and exits with status zero), then the
|
||||
script was successful.
|
||||
|
||||
Step: Verify that the required entries in /etc/hosts are present
|
||||
Step: add a verbose logging option to app.config
|
||||
Step: start three three Machi application instances
|
||||
pong
|
||||
pong
|
||||
pong
|
||||
Step: configure one chain to start a Humming Consensus group with three members
|
||||
Result: ok
|
||||
Result: ok
|
||||
Result: ok
|
||||
|
||||
We have now created a single replica chain, called `c1`, that has
|
||||
three file servers participating in the chain. Thanks to the
|
||||
hostnames that we added to `/etc/hosts`, all are using the localhost
|
||||
network interface.
|
||||
|
||||
| App instance | Pseudo | FLU name | TCP port |
|
||||
| directory | Hostname | | number |
|
||||
|--------------+----------+----------+----------|
|
||||
| dev1 | machi1 | flu1 | 20401 |
|
||||
| dev2 | machi2 | flu2 | 20402 |
|
||||
| dev3 | machi3 | flu3 | 20403 |
|
||||
|
||||
The log files for each application instance can be found in the
|
||||
`./dev/devN/log/console.log` file, where the `N` is the instance
|
||||
number: 1, 2, or 3.
|
||||
|
||||
## Understanding the chain manager's log file output
|
||||
|
||||
After running the `./priv/humming-consensus-demo.setup.sh` script,
|
||||
let's look at the last few lines of the `./dev/dev1/log/console.log`
|
||||
log file for Erlang VM process #1.
|
||||
|
||||
2016-03-09 10:16:35.676 [info] <0.105.0>@machi_lifecycle_mgr:process_pending_flu:422 Started FLU f1 with supervisor pid <0.128.0>
|
||||
2016-03-09 10:16:35.676 [info] <0.105.0>@machi_lifecycle_mgr:move_to_flu_config:540 Creating FLU config file f1
|
||||
2016-03-09 10:16:35.790 [info] <0.105.0>@machi_lifecycle_mgr:bootstrap_chain2:312 Configured chain c1 via FLU f1 to mode=ap_mode all=[f1,f2,f3] witnesses=[]
|
||||
2016-03-09 10:16:35.790 [info] <0.105.0>@machi_lifecycle_mgr:move_to_chain_config:546 Creating chain config file c1
|
||||
2016-03-09 10:16:44.139 [info] <0.132.0> CONFIRM epoch 1141 <<155,42,7,221>> upi [] rep [] auth f1 by f1
|
||||
2016-03-09 10:16:44.271 [info] <0.132.0> CONFIRM epoch 1148 <<57,213,154,16>> upi [f1] rep [] auth f1 by f1
|
||||
2016-03-09 10:16:44.864 [info] <0.132.0> CONFIRM epoch 1151 <<239,29,39,70>> upi [f1] rep [f3] auth f1 by f1
|
||||
2016-03-09 10:16:45.235 [info] <0.132.0> CONFIRM epoch 1152 <<173,17,66,225>> upi [f2] rep [f1,f3] auth f2 by f1
|
||||
2016-03-09 10:16:47.343 [info] <0.132.0> CONFIRM epoch 1154 <<154,231,224,149>> upi [f2,f1,f3] rep [] auth f2 by f1
|
||||
|
||||
Let's pick apart some of these lines. We have started all three
|
||||
servers at about the same time. We see some race conditions happen,
|
||||
and some jostling and readjustment happens pretty quickly in the first
|
||||
few seconds.
|
||||
|
||||
* `Started FLU f1 with supervisor pid <0.128.0>`
|
||||
* This VM, #1,
|
||||
started a FLU (Machi data server) with the name `f1`. In the Erlang
|
||||
process supervisor hierarchy, the process ID of the top supervisor
|
||||
is `<0.128.0>`.
|
||||
* `Configured chain c1 via FLU f1 to mode=ap_mode all=[f1,f2,f3] witnesses=[]`
|
||||
* A bootstrap configuration for a chain named `c1` has been created.
|
||||
* The FLUs/data servers that are eligible for participation in the
|
||||
chain have names `f1`, `f2`, and `f3`.
|
||||
* The chain will operate in eventual consistency mode (`ap_mode`)
|
||||
* The witness server list is empty. Witness servers are never used
|
||||
in eventual consistency mode.
|
||||
* `CONFIRM epoch 1141 <<155,42,7,221>> upi [] rep [] auth f1 by f1`
|
||||
* All participants in epoch 1141 are unanimous in adopting epoch
|
||||
1141's projection. All active membership lists are empty, so
|
||||
there is no functional chain replication yet, at least as far as
|
||||
server `f1` knows
|
||||
* The epoch's abbreviated checksum is `<<155,42,7,221>>`.
|
||||
* The UPI list, i.e. the replicas whose data is 100% in sync is
|
||||
`[]`, the empty list. (UPI = Update Propagation Invariant)
|
||||
* The list of servers that are under data repair (`rep`) is also
|
||||
empty, `[]`.
|
||||
* This projection was authored by server `f1`.
|
||||
* The log message was generated by server `f1`.
|
||||
* `CONFIRM epoch 1148 <<57,213,154,16>> upi [f1] rep [] auth f1 by f1`
|
||||
* Now the server `f1` has created a chain of length 1, `[f1]`.
|
||||
* Chain repair/file re-sync is not required when the UPI server list
|
||||
changes from length 0 -> 1.
|
||||
* `CONFIRM epoch 1151 <<239,29,39,70>> upi [f1] rep [f3] auth f1 by f1`
|
||||
* Server `f1` has noticed that server `f3` is alive. Apparently it
|
||||
has not yet noticed that server `f2` is also running.
|
||||
* Server `f3` is in the repair list.
|
||||
* `CONFIRM epoch 1152 <<173,17,66,225>> upi [f2] rep [f1,f3] auth f2 by f1`
|
||||
* Server `f2` is apparently now aware that all three servers are running.
|
||||
* The previous configuration used by `f2` was `upi [f2]`, i.e., `f2`
|
||||
was running in a chain of one. `f2` noticed that `f1` and `f3`
|
||||
were now available and has started adding them to the chain.
|
||||
* All new servers are always added to the tail of the chain in the
|
||||
repair list.
|
||||
* In eventual consistency mode, a UPI change like this is OK.
|
||||
* When performing a read, a client must read from both tail of the
|
||||
UPI list and also from all repairing servers.
|
||||
* When performing a write, the client writes to both the UPI
|
||||
server list and also the repairing list, in that order.
|
||||
* I.e., the client concatenates both lists,
|
||||
`UPI ++ Repairing`, for its chain configuration for the write.
|
||||
* Server `f2` will trigger file repair/re-sync shortly.
|
||||
* The waiting time for starting repair has been configured to be
|
||||
extremely short, 1 second. The default waiting time is 10
|
||||
seconds, in case Humming Consensus remains unstable.
|
||||
* `CONFIRM epoch 1154 <<154,231,224,149>> upi [f2,f1,f3] rep [] auth f2 by f1`
|
||||
* File repair/re-sync has finished. All file data on all servers
|
||||
are now in sync.
|
||||
* The UPI/in-sync part of the chain is now `[f2,f1,f3]`, and there
|
||||
are no servers under repair.
|
||||
|
||||
## Let's create some failures
|
||||
|
||||
Here are some suggestions for creating failures.
|
||||
|
||||
* Use the `./dev/devN/bin/machi stop` and `./dev/devN/bin/machi start`
|
||||
commands to stop & start VM #`N`.
|
||||
* Stop a VM abnormally by using `kill`. The OS process name to look
|
||||
for is `beam.smp`.
|
||||
* Suspend and resume a VM, using the `SIGSTOP` and `SIGCONT` signals.
|
||||
* E.g. `kill -STOP 9823` and `kill -CONT 9823`
|
||||
|
||||
The network partition simulator is not (yet) available when running
|
||||
Machi in this mode. Please see the next section for instructions on
|
||||
how to use partition simulator.
|
||||
|
||||
|
||||
<a name="partition-simulator">
|
||||
# Using the network partition simulator and convergence demo test code
|
||||
|
||||
This is the demo code mentioned in the presentation that Scott Lystig
|
||||
Fritchie gave at the
|
||||
[RICON 2015 conference](http://ricon.io).
|
||||
* [slides (PDF format)](http://ricon.io/speakers/slides/Scott_Fritchie_Ricon_2015.pdf)
|
||||
* [video](https://www.youtube.com/watch?v=yR5kHL1bu1Q)
|
||||
|
||||
## A complete example of all input and output
|
||||
|
||||
If you don't have an Erlang/OTP 17 runtime environment available,
|
||||
please see this file for full input and output of a strong consistency
|
||||
length=3 chain test:
|
||||
https://gist.github.com/slfritchie/8352efc88cc18e62c72c
|
||||
This file contains all commands input and all simulator output from a
|
||||
sample run of the simulator.
|
||||
|
||||
To help interpret the output of the test, please skip ahead to the
|
||||
"The test output is very verbose" section.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
If you don't have `git` and/or the Erlang 17 runtime system available
|
||||
on your OS X, FreeBSD, Linux, or Solaris machine, please take a look
|
||||
at the [Prerequisites section](#prerequisites) first. When you have
|
||||
installed the prerequisite software, please return back here.
|
||||
|
||||
## Clone and compile the code
|
||||
|
||||
Please briefly visit the [Clone and compile the code](#clone-compile)
|
||||
section. When finished, please return back here.
|
||||
|
||||
## Run an interactive Erlang CLI shell
|
||||
|
||||
Run the following command at your login shell:
|
||||
|
||||
erl -pz .eunit ebin deps/*/ebin
|
||||
|
||||
If you are using Erlang/OTP version 17, you should see some CLI output
|
||||
that looks like this:
|
||||
|
||||
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
|
||||
|
||||
Eshell V6.4 (abort with ^G)
|
||||
1>
|
||||
|
||||
## The test output is very verbose ... what are the important parts?
|
||||
|
||||
The output of the Erlang command
|
||||
`machi_chain_manager1_converge_demo:help()` will display the following
|
||||
guide to the output of the tests.
|
||||
|
||||
A visualization of the convergence behavior of the chain self-management
|
||||
algorithm for Machi.
|
||||
|
||||
1. Set up some server and chain manager pairs.
|
||||
2. Create a number of different network partition scenarios, where
|
||||
(simulated) partitions may be symmetric or asymmetric. Then stop changing
|
||||
the partitions and keep the simulated network stable (and perhaps broken).
|
||||
3. Run a number of iterations of the algorithm in parallel by poking each
|
||||
of the manager processes on a random'ish basis.
|
||||
4. Afterward, fetch the chain transition changes made by each FLU and
|
||||
verify that no transition was unsafe.
|
||||
|
||||
During the iteration periods, the following is a cheatsheet for the output.
|
||||
See the internal source for interpreting the rest of the output.
|
||||
|
||||
'SET partitions = '
|
||||
|
||||
A pair-wise list of actors which cannot send messages. The
|
||||
list is uni-directional. If there are three servers (a,b,c),
|
||||
and if the partitions list is '[{a,b},{b,c}]' then all
|
||||
messages from a->b and b->c will be dropped, but any other
|
||||
sender->recipient messages will be delivered successfully.
|
||||
|
||||
'x uses:'
|
||||
|
||||
The FLU x has made an internal state transition and is using
|
||||
this epoch's projection as operating chain configuration. The
|
||||
rest of the line is a summary of the projection.
|
||||
|
||||
'CONFIRM epoch {N}'
|
||||
|
||||
This message confirms that all of the servers listed in the
|
||||
UPI and repairing lists of the projection at epoch {N} have
|
||||
agreed to use this projection because they all have written
|
||||
this projection to their respective private projection stores.
|
||||
The chain is now usable by/available to all clients.
|
||||
|
||||
'Sweet, private projections are stable'
|
||||
|
||||
This report announces that this iteration of the test cycle
|
||||
has passed successfully. The report that follows briefly
|
||||
summarizes the latest private projection used by each
|
||||
participating server. For example, when in strong consistency
|
||||
mode with 'a' as a witness and 'b' and 'c' as real servers:
|
||||
|
||||
%% Legend:
|
||||
%% server name, epoch ID, UPI list, repairing list, down list, ...
|
||||
%% ... witness list, 'false' (a constant value)
|
||||
|
||||
[{a,{{1116,<<23,143,246,55>>},[a,b],[],[c],[a],false}},
|
||||
{b,{{1116,<<23,143,246,55>>},[a,b],[],[c],[a],false}}]
|
||||
|
||||
Both servers 'a' and 'b' agree on epoch 1116 with epoch ID
|
||||
{1116,<<23,143,246,55>>} where UPI=[a,b], repairing=[],
|
||||
down=[c], and witnesses=[a].
|
||||
|
||||
Server 'c' is not shown because 'c' has wedged itself OOS (out
|
||||
of service) by configuring a chain length of zero.
|
||||
|
||||
If no servers are listed in the report (i.e. only '[]' is
|
||||
displayed), then all servers have wedged themselves OOS, and
|
||||
the chain is unavailable.
|
||||
|
||||
'DoIt,'
|
||||
|
||||
This marks a group of tick events which trigger the manager
|
||||
processes to evaluate their environment and perhaps make a
|
||||
state transition.
|
||||
|
||||
A long chain of 'DoIt,DoIt,DoIt,' means that the chain state has
|
||||
(probably) settled to a stable configuration, which is the goal of the
|
||||
algorithm.
|
||||
|
||||
Press control-c to interrupt the test....".
|
||||
|
||||
## Run a test in eventual consistency mode
|
||||
|
||||
Run the following command at the Erlang CLI prompt:
|
||||
|
||||
machi_chain_manager1_converge_demo:t(3, [{private_write_verbose,true}]).
|
||||
|
||||
The first argument, `3`, is the number of servers to participate in
|
||||
the chain. Please note:
|
||||
|
||||
* Chain lengths as short as 1 or 2 are valid, but the results are a
|
||||
bit boring.
|
||||
* Chain lengths as long as 7 or 9 can be used, but they may
|
||||
suffer from longer periods of churn/instability before all chain
|
||||
managers reach agreement via humming consensus. (It is future work
|
||||
to shorten the worst of the unstable churn latencies.)
|
||||
* In eventual consistency mode, chain lengths may be even numbers,
|
||||
e.g. 2, 4, or 6.
|
||||
* The simulator will choose partition events from the permutations of
|
||||
all 1, 2, and 3 node partition pairs. The total runtime will
|
||||
increase *dramatically* with chain length.
|
||||
* Chain length 2: about 3 partition cases
|
||||
* Chain length 3: about 35 partition cases
|
||||
* Chain length 4: about 230 partition cases
|
||||
* Chain length 5: about 1100 partition cases
|
||||
|
||||
## Run a test in strong consistency mode (with witnesses):
|
||||
|
||||
*NOTE:* Due to a bug in the test code, please do not try to run the
|
||||
convergence test in strong consistency mode and also without the
|
||||
correct minority number of witness servers! If in doubt, please run
|
||||
the commands shown below exactly.
|
||||
|
||||
Run the following command at the Erlang CLI prompt:
|
||||
|
||||
machi_chain_manager1_converge_demo:t(3, [{private_write_verbose,true}, {consistency_mode, cp_mode}, {witnesses, [a]}]).
|
||||
|
||||
The first argument, `3`, is the number of servers to participate in
|
||||
the chain. Chain lengths as long as 7 or 9 can be used, but they may
|
||||
suffer from longer periods of churn/instability before all chain
|
||||
managers reach agreement via humming consensus.
|
||||
|
||||
Due to the bug mentioned above, please use the following
|
||||
commands when running with chain lengths of 5 or 7, respectively.
|
||||
|
||||
machi_chain_manager1_converge_demo:t(5, [{private_write_verbose,true}, {consistency_mode, cp_mode}, {witnesses, [a,b]}]).
|
||||
machi_chain_manager1_converge_demo:t(7, [{private_write_verbose,true}, {consistency_mode, cp_mode}, {witnesses, [a,b,c]}]).
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 115 KiB |
4
doc/src.high-level/.gitignore
vendored
4
doc/src.high-level/.gitignore
vendored
|
|
@ -1,4 +0,0 @@
|
|||
*.aux
|
||||
*.dvi
|
||||
*.log
|
||||
*.pdf
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
all: machi chain-mgr
|
||||
|
||||
machi:
|
||||
latex high-level-machi.tex
|
||||
dvipdfm high-level-machi.dvi
|
||||
|
||||
chain-mgr:
|
||||
latex high-level-chain-mgr.tex
|
||||
dvipdfm high-level-chain-mgr.dvi
|
||||
|
||||
clean:
|
||||
rm -f *.aux *.dvi *.log
|
||||
|
|
@ -1,268 +0,0 @@
|
|||
%!PS-Adobe-3.0 EPSF-2.0
|
||||
%%BoundingBox: 0 0 416.500000 280.000000
|
||||
%%Creator: mscgen 0.18
|
||||
%%EndComments
|
||||
0.700000 0.700000 scale
|
||||
0 0 moveto
|
||||
0 400 lineto
|
||||
595 400 lineto
|
||||
595 0 lineto
|
||||
closepath
|
||||
clip
|
||||
%PageTrailer
|
||||
%Page: 1 1
|
||||
/Helvetica findfont
|
||||
10 scalefont
|
||||
setfont
|
||||
/Helvetica findfont
|
||||
12 scalefont
|
||||
setfont
|
||||
0 400 translate
|
||||
/mtrx matrix def
|
||||
/ellipse
|
||||
{ /endangle exch def
|
||||
/startangle exch def
|
||||
/ydia exch def
|
||||
/xdia exch def
|
||||
/y exch def
|
||||
/x exch def
|
||||
/savematrix mtrx currentmatrix def
|
||||
x y translate
|
||||
xdia 2 div ydia 2 div scale
|
||||
1 -1 scale
|
||||
0 0 1 startangle endangle arc
|
||||
savematrix setmatrix
|
||||
} def
|
||||
(client) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 42 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
42 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(Projection) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 127 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
127 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(ProjStore_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 212 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
212 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(Sequencer_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 297 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
297 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 382 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
382 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_B) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 467 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
467 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_C) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 552 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
552 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
newpath 42 -22 moveto 42 -49 lineto stroke
|
||||
newpath 127 -22 moveto 127 -49 lineto stroke
|
||||
newpath 212 -22 moveto 212 -49 lineto stroke
|
||||
newpath 297 -22 moveto 297 -49 lineto stroke
|
||||
newpath 382 -22 moveto 382 -49 lineto stroke
|
||||
newpath 467 -22 moveto 467 -49 lineto stroke
|
||||
newpath 552 -22 moveto 552 -49 lineto stroke
|
||||
newpath 42 -35 moveto 127 -35 lineto stroke
|
||||
newpath 127 -35 moveto 117 -41 lineto stroke
|
||||
(get current) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 57 -33 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
57 -33 moveto show
|
||||
newpath 42 -49 moveto 42 -76 lineto stroke
|
||||
newpath 127 -49 moveto 127 -76 lineto stroke
|
||||
newpath 212 -49 moveto 212 -76 lineto stroke
|
||||
newpath 297 -49 moveto 297 -76 lineto stroke
|
||||
newpath 382 -49 moveto 382 -76 lineto stroke
|
||||
newpath 467 -49 moveto 467 -76 lineto stroke
|
||||
newpath 552 -49 moveto 552 -76 lineto stroke
|
||||
newpath 127 -62 moveto 42 -62 lineto stroke
|
||||
newpath 42 -62 moveto 52 -68 lineto stroke
|
||||
(ok, #12...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 61 -60 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
61 -60 moveto show
|
||||
newpath 42 -76 moveto 42 -103 lineto stroke
|
||||
newpath 127 -76 moveto 127 -103 lineto stroke
|
||||
newpath 212 -76 moveto 212 -103 lineto stroke
|
||||
newpath 297 -76 moveto 297 -103 lineto stroke
|
||||
newpath 382 -76 moveto 382 -103 lineto stroke
|
||||
newpath 467 -76 moveto 467 -103 lineto stroke
|
||||
newpath 552 -76 moveto 552 -103 lineto stroke
|
||||
newpath 42 -89 moveto 297 -89 lineto stroke
|
||||
newpath 297 -89 moveto 287 -95 lineto stroke
|
||||
(Req. 123 bytes, prefix="foo", epoch=12) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 66 -87 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
66 -87 moveto show
|
||||
newpath 42 -103 moveto 42 -130 lineto stroke
|
||||
newpath 127 -103 moveto 127 -130 lineto stroke
|
||||
newpath 212 -103 moveto 212 -130 lineto stroke
|
||||
newpath 297 -103 moveto 297 -130 lineto stroke
|
||||
newpath 382 -103 moveto 382 -130 lineto stroke
|
||||
newpath 467 -103 moveto 467 -130 lineto stroke
|
||||
newpath 552 -103 moveto 552 -130 lineto stroke
|
||||
newpath 297 -116 moveto 42 -116 lineto stroke
|
||||
newpath 42 -116 moveto 52 -122 lineto stroke
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
(bad_epoch, 13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 131 -114 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
131 -114 moveto show
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
newpath 42 -130 moveto 42 -157 lineto stroke
|
||||
newpath 127 -130 moveto 127 -157 lineto stroke
|
||||
newpath 212 -130 moveto 212 -157 lineto stroke
|
||||
newpath 297 -130 moveto 297 -157 lineto stroke
|
||||
newpath 382 -130 moveto 382 -157 lineto stroke
|
||||
newpath 467 -130 moveto 467 -157 lineto stroke
|
||||
newpath 552 -130 moveto 552 -157 lineto stroke
|
||||
newpath 42 -143 moveto 212 -143 lineto stroke
|
||||
newpath 212 -143 moveto 202 -149 lineto stroke
|
||||
(get epoch #13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 89 -141 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
89 -141 moveto show
|
||||
newpath 42 -157 moveto 42 -184 lineto stroke
|
||||
newpath 127 -157 moveto 127 -184 lineto stroke
|
||||
newpath 212 -157 moveto 212 -184 lineto stroke
|
||||
newpath 297 -157 moveto 297 -184 lineto stroke
|
||||
newpath 382 -157 moveto 382 -184 lineto stroke
|
||||
newpath 467 -157 moveto 467 -184 lineto stroke
|
||||
newpath 552 -157 moveto 552 -184 lineto stroke
|
||||
newpath 212 -170 moveto 42 -170 lineto stroke
|
||||
newpath 42 -170 moveto 52 -176 lineto stroke
|
||||
(ok, #13...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 103 -168 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
103 -168 moveto show
|
||||
newpath 42 -184 moveto 42 -211 lineto stroke
|
||||
newpath 127 -184 moveto 127 -211 lineto stroke
|
||||
newpath 212 -184 moveto 212 -211 lineto stroke
|
||||
newpath 297 -184 moveto 297 -211 lineto stroke
|
||||
newpath 382 -184 moveto 382 -211 lineto stroke
|
||||
newpath 467 -184 moveto 467 -211 lineto stroke
|
||||
newpath 552 -184 moveto 552 -211 lineto stroke
|
||||
newpath 42 -197 moveto 297 -197 lineto stroke
|
||||
newpath 297 -197 moveto 287 -203 lineto stroke
|
||||
(Req. 123 bytes, prefix="foo", epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 66 -195 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
66 -195 moveto show
|
||||
newpath 42 -211 moveto 42 -238 lineto stroke
|
||||
newpath 127 -211 moveto 127 -238 lineto stroke
|
||||
newpath 212 -211 moveto 212 -238 lineto stroke
|
||||
newpath 297 -211 moveto 297 -238 lineto stroke
|
||||
newpath 382 -211 moveto 382 -238 lineto stroke
|
||||
newpath 467 -211 moveto 467 -238 lineto stroke
|
||||
newpath 552 -211 moveto 552 -238 lineto stroke
|
||||
newpath 297 -224 moveto 42 -224 lineto stroke
|
||||
newpath 42 -224 moveto 52 -230 lineto stroke
|
||||
(ok, "foo.seq_a.009" offset=447) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 89 -222 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
89 -222 moveto show
|
||||
newpath 42 -238 moveto 42 -265 lineto stroke
|
||||
newpath 127 -238 moveto 127 -265 lineto stroke
|
||||
newpath 212 -238 moveto 212 -265 lineto stroke
|
||||
newpath 297 -238 moveto 297 -265 lineto stroke
|
||||
newpath 382 -238 moveto 382 -265 lineto stroke
|
||||
newpath 467 -238 moveto 467 -265 lineto stroke
|
||||
newpath 552 -238 moveto 552 -265 lineto stroke
|
||||
newpath 42 -251 moveto 382 -251 lineto stroke
|
||||
newpath 382 -251 moveto 372 -257 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 51 -249 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
51 -249 moveto show
|
||||
newpath 42 -265 moveto 42 -292 lineto stroke
|
||||
newpath 127 -265 moveto 127 -292 lineto stroke
|
||||
newpath 212 -265 moveto 212 -292 lineto stroke
|
||||
newpath 297 -265 moveto 297 -292 lineto stroke
|
||||
newpath 382 -265 moveto 382 -292 lineto stroke
|
||||
newpath 467 -265 moveto 467 -292 lineto stroke
|
||||
newpath 552 -265 moveto 552 -292 lineto stroke
|
||||
newpath 382 -278 moveto 42 -278 lineto stroke
|
||||
newpath 42 -278 moveto 52 -284 lineto stroke
|
||||
(ok) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 206 -276 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
206 -276 moveto show
|
||||
newpath 42 -292 moveto 42 -319 lineto stroke
|
||||
newpath 127 -292 moveto 127 -319 lineto stroke
|
||||
newpath 212 -292 moveto 212 -319 lineto stroke
|
||||
newpath 297 -292 moveto 297 -319 lineto stroke
|
||||
newpath 382 -292 moveto 382 -319 lineto stroke
|
||||
newpath 467 -292 moveto 467 -319 lineto stroke
|
||||
newpath 552 -292 moveto 552 -319 lineto stroke
|
||||
newpath 42 -305 moveto 467 -305 lineto stroke
|
||||
newpath 467 -305 moveto 457 -311 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 94 -303 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
94 -303 moveto show
|
||||
newpath 42 -319 moveto 42 -346 lineto stroke
|
||||
newpath 127 -319 moveto 127 -346 lineto stroke
|
||||
newpath 212 -319 moveto 212 -346 lineto stroke
|
||||
newpath 297 -319 moveto 297 -346 lineto stroke
|
||||
newpath 382 -319 moveto 382 -346 lineto stroke
|
||||
newpath 467 -319 moveto 467 -346 lineto stroke
|
||||
newpath 552 -319 moveto 552 -346 lineto stroke
|
||||
newpath 467 -332 moveto 42 -332 lineto stroke
|
||||
newpath 42 -332 moveto 52 -338 lineto stroke
|
||||
(ok) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 249 -330 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
249 -330 moveto show
|
||||
newpath 42 -346 moveto 42 -373 lineto stroke
|
||||
newpath 127 -346 moveto 127 -373 lineto stroke
|
||||
newpath 212 -346 moveto 212 -373 lineto stroke
|
||||
newpath 297 -346 moveto 297 -373 lineto stroke
|
||||
newpath 382 -346 moveto 382 -373 lineto stroke
|
||||
newpath 467 -346 moveto 467 -373 lineto stroke
|
||||
newpath 552 -346 moveto 552 -373 lineto stroke
|
||||
newpath 42 -359 moveto 552 -359 lineto stroke
|
||||
newpath 552 -359 moveto 542 -365 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 136 -357 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
136 -357 moveto show
|
||||
newpath 42 -373 moveto 42 -400 lineto stroke
|
||||
newpath 127 -373 moveto 127 -400 lineto stroke
|
||||
newpath 212 -373 moveto 212 -400 lineto stroke
|
||||
newpath 297 -373 moveto 297 -400 lineto stroke
|
||||
newpath 382 -373 moveto 382 -400 lineto stroke
|
||||
newpath 467 -373 moveto 467 -400 lineto stroke
|
||||
newpath 552 -373 moveto 552 -400 lineto stroke
|
||||
newpath 552 -386 moveto 42 -386 lineto stroke
|
||||
newpath 42 -386 moveto 52 -392 lineto stroke
|
||||
(ok) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 291 -384 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
291 -384 moveto show
|
||||
|
|
@ -1,345 +0,0 @@
|
|||
%!PS-Adobe-3.0 EPSF-2.0
|
||||
%%BoundingBox: 0 0 416.500000 355.600006
|
||||
%%Creator: mscgen 0.18
|
||||
%%EndComments
|
||||
0.700000 0.700000 scale
|
||||
0 0 moveto
|
||||
0 508 lineto
|
||||
595 508 lineto
|
||||
595 0 lineto
|
||||
closepath
|
||||
clip
|
||||
%PageTrailer
|
||||
%Page: 1 1
|
||||
/Helvetica findfont
|
||||
10 scalefont
|
||||
setfont
|
||||
/Helvetica findfont
|
||||
12 scalefont
|
||||
setfont
|
||||
0 508 translate
|
||||
/mtrx matrix def
|
||||
/ellipse
|
||||
{ /endangle exch def
|
||||
/startangle exch def
|
||||
/ydia exch def
|
||||
/xdia exch def
|
||||
/y exch def
|
||||
/x exch def
|
||||
/savematrix mtrx currentmatrix def
|
||||
x y translate
|
||||
xdia 2 div ydia 2 div scale
|
||||
1 -1 scale
|
||||
0 0 1 startangle endangle arc
|
||||
savematrix setmatrix
|
||||
} def
|
||||
(client) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 42 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
42 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(Projection) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 127 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
127 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(ProjStore_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 212 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
212 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(Sequencer_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 297 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
297 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_A) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 382 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
382 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_B) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 467 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
467 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_C) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 552 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
552 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
newpath 42 -22 moveto 42 -49 lineto stroke
|
||||
newpath 127 -22 moveto 127 -49 lineto stroke
|
||||
newpath 212 -22 moveto 212 -49 lineto stroke
|
||||
newpath 297 -22 moveto 297 -49 lineto stroke
|
||||
newpath 382 -22 moveto 382 -49 lineto stroke
|
||||
newpath 467 -22 moveto 467 -49 lineto stroke
|
||||
newpath 552 -22 moveto 552 -49 lineto stroke
|
||||
newpath 42 -35 moveto 127 -35 lineto stroke
|
||||
newpath 127 -35 moveto 117 -41 lineto stroke
|
||||
(get current) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 57 -33 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
57 -33 moveto show
|
||||
newpath 42 -49 moveto 42 -76 lineto stroke
|
||||
newpath 127 -49 moveto 127 -76 lineto stroke
|
||||
newpath 212 -49 moveto 212 -76 lineto stroke
|
||||
newpath 297 -49 moveto 297 -76 lineto stroke
|
||||
newpath 382 -49 moveto 382 -76 lineto stroke
|
||||
newpath 467 -49 moveto 467 -76 lineto stroke
|
||||
newpath 552 -49 moveto 552 -76 lineto stroke
|
||||
newpath 127 -62 moveto 42 -62 lineto stroke
|
||||
newpath 42 -62 moveto 52 -68 lineto stroke
|
||||
(ok, #12...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 61 -60 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
61 -60 moveto show
|
||||
newpath 42 -76 moveto 42 -103 lineto stroke
|
||||
newpath 127 -76 moveto 127 -103 lineto stroke
|
||||
newpath 212 -76 moveto 212 -103 lineto stroke
|
||||
newpath 297 -76 moveto 297 -103 lineto stroke
|
||||
newpath 382 -76 moveto 382 -103 lineto stroke
|
||||
newpath 467 -76 moveto 467 -103 lineto stroke
|
||||
newpath 552 -76 moveto 552 -103 lineto stroke
|
||||
newpath 42 -89 moveto 382 -89 lineto stroke
|
||||
newpath 382 -89 moveto 372 -95 lineto stroke
|
||||
(append prefix="foo" <<123 bytes...>> epoch=12) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 85 -87 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
85 -87 moveto show
|
||||
newpath 42 -103 moveto 42 -130 lineto stroke
|
||||
newpath 127 -103 moveto 127 -130 lineto stroke
|
||||
newpath 212 -103 moveto 212 -130 lineto stroke
|
||||
newpath 297 -103 moveto 297 -130 lineto stroke
|
||||
newpath 382 -103 moveto 382 -130 lineto stroke
|
||||
newpath 467 -103 moveto 467 -130 lineto stroke
|
||||
newpath 552 -103 moveto 552 -130 lineto stroke
|
||||
newpath 382 -116 moveto 42 -116 lineto stroke
|
||||
newpath 42 -116 moveto 52 -122 lineto stroke
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
(bad_epoch, 13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 173 -114 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
173 -114 moveto show
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
newpath 42 -130 moveto 42 -157 lineto stroke
|
||||
newpath 127 -130 moveto 127 -157 lineto stroke
|
||||
newpath 212 -130 moveto 212 -157 lineto stroke
|
||||
newpath 297 -130 moveto 297 -157 lineto stroke
|
||||
newpath 382 -130 moveto 382 -157 lineto stroke
|
||||
newpath 467 -130 moveto 467 -157 lineto stroke
|
||||
newpath 552 -130 moveto 552 -157 lineto stroke
|
||||
newpath 42 -143 moveto 212 -143 lineto stroke
|
||||
newpath 212 -143 moveto 202 -149 lineto stroke
|
||||
(get epoch #13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 89 -141 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
89 -141 moveto show
|
||||
newpath 42 -157 moveto 42 -184 lineto stroke
|
||||
newpath 127 -157 moveto 127 -184 lineto stroke
|
||||
newpath 212 -157 moveto 212 -184 lineto stroke
|
||||
newpath 297 -157 moveto 297 -184 lineto stroke
|
||||
newpath 382 -157 moveto 382 -184 lineto stroke
|
||||
newpath 467 -157 moveto 467 -184 lineto stroke
|
||||
newpath 552 -157 moveto 552 -184 lineto stroke
|
||||
newpath 212 -170 moveto 42 -170 lineto stroke
|
||||
newpath 42 -170 moveto 52 -176 lineto stroke
|
||||
(ok, #13...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 103 -168 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
103 -168 moveto show
|
||||
newpath 42 -184 moveto 42 -211 lineto stroke
|
||||
newpath 127 -184 moveto 127 -211 lineto stroke
|
||||
newpath 212 -184 moveto 212 -211 lineto stroke
|
||||
newpath 297 -184 moveto 297 -211 lineto stroke
|
||||
newpath 382 -184 moveto 382 -211 lineto stroke
|
||||
newpath 467 -184 moveto 467 -211 lineto stroke
|
||||
newpath 552 -184 moveto 552 -211 lineto stroke
|
||||
newpath 42 -197 moveto 382 -197 lineto stroke
|
||||
newpath 382 -197 moveto 372 -203 lineto stroke
|
||||
(append prefix="foo" <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 85 -195 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
85 -195 moveto show
|
||||
newpath 42 -211 moveto 42 -238 lineto stroke
|
||||
newpath 127 -211 moveto 127 -238 lineto stroke
|
||||
newpath 212 -211 moveto 212 -238 lineto stroke
|
||||
newpath 297 -211 moveto 297 -238 lineto stroke
|
||||
newpath 382 -211 moveto 382 -238 lineto stroke
|
||||
newpath 467 -211 moveto 467 -238 lineto stroke
|
||||
newpath 552 -211 moveto 552 -238 lineto stroke
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
newpath 263 -211 moveto 417 -211 lineto 417 -236 lineto 263 -236 lineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
newpath 269 -211 moveto 411 -211 lineto stroke
|
||||
newpath 269 -236 moveto 411 -236 lineto stroke
|
||||
newpath 269 -211 moveto 263 -223 lineto stroke
|
||||
newpath 263 -223 moveto 269 -236 lineto stroke
|
||||
newpath 411 -211 moveto 417 -223 lineto stroke
|
||||
newpath 417 -223 moveto 411 -236 lineto stroke
|
||||
(Co-located on same box) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 275 -227 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
275 -227 moveto show
|
||||
newpath 42 -238 moveto 42 -265 lineto stroke
|
||||
newpath 127 -238 moveto 127 -265 lineto stroke
|
||||
newpath 212 -238 moveto 212 -265 lineto stroke
|
||||
newpath 297 -238 moveto 297 -265 lineto stroke
|
||||
newpath 382 -238 moveto 382 -265 lineto stroke
|
||||
newpath 467 -238 moveto 467 -265 lineto stroke
|
||||
newpath 552 -238 moveto 552 -265 lineto stroke
|
||||
newpath 382 -251 moveto 297 -251 lineto stroke
|
||||
newpath 297 -251 moveto 307 -257 lineto stroke
|
||||
(Req. 123 bytes, prefix="foo", epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 236 -249 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
236 -249 moveto show
|
||||
newpath 42 -265 moveto 42 -292 lineto stroke
|
||||
newpath 127 -265 moveto 127 -292 lineto stroke
|
||||
newpath 212 -265 moveto 212 -292 lineto stroke
|
||||
newpath 297 -265 moveto 297 -292 lineto stroke
|
||||
newpath 382 -265 moveto 382 -292 lineto stroke
|
||||
newpath 467 -265 moveto 467 -292 lineto stroke
|
||||
newpath 552 -265 moveto 552 -292 lineto stroke
|
||||
newpath 297 -278 moveto 382 -278 lineto stroke
|
||||
newpath 382 -278 moveto 372 -284 lineto stroke
|
||||
(ok, "foo.seq_a.009" offset=447) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 259 -276 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
259 -276 moveto show
|
||||
newpath 42 -292 moveto 42 -319 lineto stroke
|
||||
newpath 127 -292 moveto 127 -319 lineto stroke
|
||||
newpath 212 -292 moveto 212 -319 lineto stroke
|
||||
newpath 297 -292 moveto 297 -319 lineto stroke
|
||||
newpath 382 -292 moveto 382 -319 lineto stroke
|
||||
newpath 467 -292 moveto 467 -319 lineto stroke
|
||||
newpath 552 -292 moveto 552 -319 lineto stroke
|
||||
newpath 382 -305 85 13 270 90 ellipse stroke
|
||||
newpath 382 -311 moveto 392 -317 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 58 -303 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
58 -303 moveto show
|
||||
newpath 42 -319 moveto 42 -346 lineto stroke
|
||||
newpath 127 -319 moveto 127 -346 lineto stroke
|
||||
newpath 212 -319 moveto 212 -346 lineto stroke
|
||||
newpath 297 -319 moveto 297 -346 lineto stroke
|
||||
newpath 382 -319 moveto 382 -346 lineto stroke
|
||||
newpath 467 -319 moveto 467 -346 lineto stroke
|
||||
newpath 552 -319 moveto 552 -346 lineto stroke
|
||||
newpath 382 -332 moveto 467 -332 lineto stroke
|
||||
newpath 467 -332 moveto 457 -338 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 264 -330 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
264 -330 moveto show
|
||||
newpath 42 -346 moveto 42 -373 lineto stroke
|
||||
newpath 127 -346 moveto 127 -373 lineto stroke
|
||||
newpath 212 -346 moveto 212 -373 lineto stroke
|
||||
newpath 297 -346 moveto 297 -373 lineto stroke
|
||||
newpath 382 -346 moveto 382 -373 lineto stroke
|
||||
newpath 467 -346 moveto 467 -373 lineto stroke
|
||||
newpath 552 -346 moveto 552 -373 lineto stroke
|
||||
newpath 467 -359 moveto 552 -359 lineto stroke
|
||||
newpath 552 -359 moveto 542 -365 lineto stroke
|
||||
(write "foo.seq_a.009" offset=447 <<123 bytes...>> epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 273 -357 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
273 -357 moveto show
|
||||
newpath 42 -373 moveto 42 -400 lineto stroke
|
||||
newpath 127 -373 moveto 127 -400 lineto stroke
|
||||
newpath 212 -373 moveto 212 -400 lineto stroke
|
||||
newpath 297 -373 moveto 297 -400 lineto stroke
|
||||
newpath 382 -373 moveto 382 -400 lineto stroke
|
||||
newpath 467 -373 moveto 467 -400 lineto stroke
|
||||
newpath 552 -373 moveto 552 -400 lineto stroke
|
||||
newpath 552 -386 moveto 42 -386 lineto stroke
|
||||
newpath 42 -386 moveto 52 -392 lineto stroke
|
||||
(ok, "foo.seq_a.009" offset=447) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 216 -384 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
216 -384 moveto show
|
||||
newpath 42 -400 moveto 42 -427 lineto stroke
|
||||
newpath 127 -400 moveto 127 -427 lineto stroke
|
||||
newpath 212 -400 moveto 212 -427 lineto stroke
|
||||
newpath 297 -400 moveto 297 -427 lineto stroke
|
||||
newpath 382 -400 moveto 382 -427 lineto stroke
|
||||
newpath 467 -400 moveto 467 -427 lineto stroke
|
||||
newpath 552 -400 moveto 552 -427 lineto stroke
|
||||
(The above is "fast path" for FLU->FLU forwarding.) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 164 -416 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
164 -416 moveto show
|
||||
[2] 0 setdash
|
||||
newpath 21 -413 moveto 162 -413 lineto stroke
|
||||
[] 0 setdash
|
||||
[2] 0 setdash
|
||||
newpath 432 -413 moveto 574 -413 lineto stroke
|
||||
[] 0 setdash
|
||||
newpath 42 -427 moveto 42 -454 lineto stroke
|
||||
newpath 127 -427 moveto 127 -454 lineto stroke
|
||||
newpath 212 -427 moveto 212 -454 lineto stroke
|
||||
newpath 297 -427 moveto 297 -454 lineto stroke
|
||||
newpath 382 -427 moveto 382 -454 lineto stroke
|
||||
newpath 467 -427 moveto 467 -454 lineto stroke
|
||||
newpath 552 -427 moveto 552 -454 lineto stroke
|
||||
(If, in an alternate scenario, FLU_C has an error...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 167 -443 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
167 -443 moveto show
|
||||
[2] 0 setdash
|
||||
newpath 21 -440 moveto 165 -440 lineto stroke
|
||||
[] 0 setdash
|
||||
[2] 0 setdash
|
||||
newpath 429 -440 moveto 574 -440 lineto stroke
|
||||
[] 0 setdash
|
||||
newpath 42 -454 moveto 42 -481 lineto stroke
|
||||
newpath 127 -454 moveto 127 -481 lineto stroke
|
||||
newpath 212 -454 moveto 212 -481 lineto stroke
|
||||
newpath 297 -454 moveto 297 -481 lineto stroke
|
||||
newpath 382 -454 moveto 382 -481 lineto stroke
|
||||
newpath 467 -454 moveto 467 -481 lineto stroke
|
||||
newpath 552 -454 moveto 552 -481 lineto stroke
|
||||
newpath 552 -467 moveto 42 -467 lineto stroke
|
||||
newpath 42 -467 moveto 52 -473 lineto stroke
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
(bad_epoch, 15) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 258 -465 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
258 -465 moveto show
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
newpath 42 -481 moveto 42 -508 lineto stroke
|
||||
newpath 127 -481 moveto 127 -508 lineto stroke
|
||||
newpath 212 -481 moveto 212 -508 lineto stroke
|
||||
newpath 297 -481 moveto 297 -508 lineto stroke
|
||||
newpath 382 -481 moveto 382 -508 lineto stroke
|
||||
newpath 467 -481 moveto 467 -508 lineto stroke
|
||||
newpath 552 -481 moveto 552 -508 lineto stroke
|
||||
(... then repair becomes the client's responsibility \("slow path"\).) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 133 -497 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
133 -497 moveto show
|
||||
[2] 0 setdash
|
||||
newpath 21 -494 moveto 131 -494 lineto stroke
|
||||
[] 0 setdash
|
||||
[2] 0 setdash
|
||||
newpath 464 -494 moveto 574 -494 lineto stroke
|
||||
[] 0 setdash
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,557 +0,0 @@
|
|||
%!PS-Adobe-3.0 EPSF-3.0
|
||||
%%Title: figure6.fig
|
||||
%%Creator: fig2dev Version 3.2 Patchlevel 5d
|
||||
%%CreationDate: Mon Oct 20 21:56:33 2014
|
||||
%%For: fritchie@sbb3.local (Scott Lystig Fritchie)
|
||||
%%BoundingBox: 0 0 633 332
|
||||
%Magnification: 1.0000
|
||||
%%EndComments
|
||||
%%BeginProlog
|
||||
/$F2psDict 200 dict def
|
||||
$F2psDict begin
|
||||
$F2psDict /mtrx matrix put
|
||||
/col-1 {0 setgray} bind def
|
||||
/col0 {0.000 0.000 0.000 srgb} bind def
|
||||
/col1 {0.000 0.000 1.000 srgb} bind def
|
||||
/col2 {0.000 1.000 0.000 srgb} bind def
|
||||
/col3 {0.000 1.000 1.000 srgb} bind def
|
||||
/col4 {1.000 0.000 0.000 srgb} bind def
|
||||
/col5 {1.000 0.000 1.000 srgb} bind def
|
||||
/col6 {1.000 1.000 0.000 srgb} bind def
|
||||
/col7 {1.000 1.000 1.000 srgb} bind def
|
||||
/col8 {0.000 0.000 0.560 srgb} bind def
|
||||
/col9 {0.000 0.000 0.690 srgb} bind def
|
||||
/col10 {0.000 0.000 0.820 srgb} bind def
|
||||
/col11 {0.530 0.810 1.000 srgb} bind def
|
||||
/col12 {0.000 0.560 0.000 srgb} bind def
|
||||
/col13 {0.000 0.690 0.000 srgb} bind def
|
||||
/col14 {0.000 0.820 0.000 srgb} bind def
|
||||
/col15 {0.000 0.560 0.560 srgb} bind def
|
||||
/col16 {0.000 0.690 0.690 srgb} bind def
|
||||
/col17 {0.000 0.820 0.820 srgb} bind def
|
||||
/col18 {0.560 0.000 0.000 srgb} bind def
|
||||
/col19 {0.690 0.000 0.000 srgb} bind def
|
||||
/col20 {0.820 0.000 0.000 srgb} bind def
|
||||
/col21 {0.560 0.000 0.560 srgb} bind def
|
||||
/col22 {0.690 0.000 0.690 srgb} bind def
|
||||
/col23 {0.820 0.000 0.820 srgb} bind def
|
||||
/col24 {0.500 0.190 0.000 srgb} bind def
|
||||
/col25 {0.630 0.250 0.000 srgb} bind def
|
||||
/col26 {0.750 0.380 0.000 srgb} bind def
|
||||
/col27 {1.000 0.500 0.500 srgb} bind def
|
||||
/col28 {1.000 0.630 0.630 srgb} bind def
|
||||
/col29 {1.000 0.750 0.750 srgb} bind def
|
||||
/col30 {1.000 0.880 0.880 srgb} bind def
|
||||
/col31 {1.000 0.840 0.000 srgb} bind def
|
||||
|
||||
end
|
||||
|
||||
/cp {closepath} bind def
|
||||
/ef {eofill} bind def
|
||||
/gr {grestore} bind def
|
||||
/gs {gsave} bind def
|
||||
/sa {save} bind def
|
||||
/rs {restore} bind def
|
||||
/l {lineto} bind def
|
||||
/m {moveto} bind def
|
||||
/rm {rmoveto} bind def
|
||||
/n {newpath} bind def
|
||||
/s {stroke} bind def
|
||||
/sh {show} bind def
|
||||
/slc {setlinecap} bind def
|
||||
/slj {setlinejoin} bind def
|
||||
/slw {setlinewidth} bind def
|
||||
/srgb {setrgbcolor} bind def
|
||||
/rot {rotate} bind def
|
||||
/sc {scale} bind def
|
||||
/sd {setdash} bind def
|
||||
/ff {findfont} bind def
|
||||
/sf {setfont} bind def
|
||||
/scf {scalefont} bind def
|
||||
/sw {stringwidth} bind def
|
||||
/tr {translate} bind def
|
||||
/tnt {dup dup currentrgbcolor
|
||||
4 -2 roll dup 1 exch sub 3 -1 roll mul add
|
||||
4 -2 roll dup 1 exch sub 3 -1 roll mul add
|
||||
4 -2 roll dup 1 exch sub 3 -1 roll mul add srgb}
|
||||
bind def
|
||||
/shd {dup dup currentrgbcolor 4 -2 roll mul 4 -2 roll mul
|
||||
4 -2 roll mul srgb} bind def
|
||||
/reencdict 12 dict def /ReEncode { reencdict begin
|
||||
/newcodesandnames exch def /newfontname exch def /basefontname exch def
|
||||
/basefontdict basefontname findfont def /newfont basefontdict maxlength dict def
|
||||
basefontdict { exch dup /FID ne { dup /Encoding eq
|
||||
{ exch dup length array copy newfont 3 1 roll put }
|
||||
{ exch newfont 3 1 roll put } ifelse } { pop pop } ifelse } forall
|
||||
newfont /FontName newfontname put newcodesandnames aload pop
|
||||
128 1 255 { newfont /Encoding get exch /.notdef put } for
|
||||
newcodesandnames length 2 idiv { newfont /Encoding get 3 1 roll put } repeat
|
||||
newfontname newfont definefont pop end } def
|
||||
/isovec [
|
||||
8#055 /minus 8#200 /grave 8#201 /acute 8#202 /circumflex 8#203 /tilde
|
||||
8#204 /macron 8#205 /breve 8#206 /dotaccent 8#207 /dieresis
|
||||
8#210 /ring 8#211 /cedilla 8#212 /hungarumlaut 8#213 /ogonek 8#214 /caron
|
||||
8#220 /dotlessi 8#230 /oe 8#231 /OE
|
||||
8#240 /space 8#241 /exclamdown 8#242 /cent 8#243 /sterling
|
||||
8#244 /currency 8#245 /yen 8#246 /brokenbar 8#247 /section 8#250 /dieresis
|
||||
8#251 /copyright 8#252 /ordfeminine 8#253 /guillemotleft 8#254 /logicalnot
|
||||
8#255 /hyphen 8#256 /registered 8#257 /macron 8#260 /degree 8#261 /plusminus
|
||||
8#262 /twosuperior 8#263 /threesuperior 8#264 /acute 8#265 /mu 8#266 /paragraph
|
||||
8#267 /periodcentered 8#270 /cedilla 8#271 /onesuperior 8#272 /ordmasculine
|
||||
8#273 /guillemotright 8#274 /onequarter 8#275 /onehalf
|
||||
8#276 /threequarters 8#277 /questiondown 8#300 /Agrave 8#301 /Aacute
|
||||
8#302 /Acircumflex 8#303 /Atilde 8#304 /Adieresis 8#305 /Aring
|
||||
8#306 /AE 8#307 /Ccedilla 8#310 /Egrave 8#311 /Eacute
|
||||
8#312 /Ecircumflex 8#313 /Edieresis 8#314 /Igrave 8#315 /Iacute
|
||||
8#316 /Icircumflex 8#317 /Idieresis 8#320 /Eth 8#321 /Ntilde 8#322 /Ograve
|
||||
8#323 /Oacute 8#324 /Ocircumflex 8#325 /Otilde 8#326 /Odieresis 8#327 /multiply
|
||||
8#330 /Oslash 8#331 /Ugrave 8#332 /Uacute 8#333 /Ucircumflex
|
||||
8#334 /Udieresis 8#335 /Yacute 8#336 /Thorn 8#337 /germandbls 8#340 /agrave
|
||||
8#341 /aacute 8#342 /acircumflex 8#343 /atilde 8#344 /adieresis 8#345 /aring
|
||||
8#346 /ae 8#347 /ccedilla 8#350 /egrave 8#351 /eacute
|
||||
8#352 /ecircumflex 8#353 /edieresis 8#354 /igrave 8#355 /iacute
|
||||
8#356 /icircumflex 8#357 /idieresis 8#360 /eth 8#361 /ntilde 8#362 /ograve
|
||||
8#363 /oacute 8#364 /ocircumflex 8#365 /otilde 8#366 /odieresis 8#367 /divide
|
||||
8#370 /oslash 8#371 /ugrave 8#372 /uacute 8#373 /ucircumflex
|
||||
8#374 /udieresis 8#375 /yacute 8#376 /thorn 8#377 /ydieresis] def
|
||||
/Times-Bold /Times-Bold-iso isovec ReEncode
|
||||
/Times-Roman /Times-Roman-iso isovec ReEncode
|
||||
/$F2psBegin {$F2psDict begin /$F2psEnteredState save def} def
|
||||
/$F2psEnd {$F2psEnteredState restore end} def
|
||||
|
||||
/pageheader {
|
||||
save
|
||||
newpath 0 332 moveto 0 0 lineto 633 0 lineto 633 332 lineto closepath clip newpath
|
||||
-331.7 473.8 translate
|
||||
1 -1 scale
|
||||
$F2psBegin
|
||||
10 setmiterlimit
|
||||
0 slj 0 slc
|
||||
0.06000 0.06000 sc
|
||||
} bind def
|
||||
/pagefooter {
|
||||
$F2psEnd
|
||||
restore
|
||||
} bind def
|
||||
%%EndProlog
|
||||
pageheader
|
||||
%
|
||||
% Fig objects follow
|
||||
%
|
||||
%
|
||||
% here starts figure with depth 50
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
5925 7275 m
|
||||
gs 1 -1 sc (Step 6: Client now knows that projection 12 is invalid. Fetch projection 13, then retry at step #8.) col16 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8550 3225 m
|
||||
gs 1 -1 sc (Get epoch 13) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 6900 m
|
||||
gs 1 -1 sc (Active=[a,b,c]) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 6675 m
|
||||
gs 1 -1 sc (Members=[a,b,c]) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 6450 m
|
||||
gs 1 -1 sc (Epoch=13) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 5580 m
|
||||
gs 1 -1 sc (Epoch=12) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 5835 m
|
||||
gs 1 -1 sc (Members=[a,b,c]) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 6090 m
|
||||
gs 1 -1 sc (Active=[a,b]) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5925 5175 m
|
||||
gs 1 -1 sc (Projection \(data structure\)) col0 sh gr
|
||||
% Polyline
|
||||
0 slj
|
||||
0 slc
|
||||
15.000 slw
|
||||
n 8400 4950 m 5625 4950 l 5625 7050 l 8400 7050 l
|
||||
cp gs col0 s gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 6405 m
|
||||
gs 1 -1 sc (- write once) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 6660 m
|
||||
gs 1 -1 sc (- key=integer) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 6915 m
|
||||
gs 1 -1 sc (- value=projection data structure) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 7170 m
|
||||
gs 1 -1 sc (k=11, v=...) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 7425 m
|
||||
gs 1 -1 sc (k=12, v=...) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 7680 m
|
||||
gs 1 -1 sc (k=13, v=...) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 6150 m
|
||||
gs 1 -1 sc (FLU projection store \(proc\)) col0 sh gr
|
||||
% Polyline
|
||||
n 12750 5925 m 15900 5925 l 15900 7725 l 12750 7725 l
|
||||
cp gs col0 s gr
|
||||
% Polyline
|
||||
gs clippath
|
||||
14788 5055 m 14940 5055 l 14940 4995 l 14788 4995 l 14788 4995 l 14908 5025 l 14788 5055 l cp
|
||||
14612 4995 m 14460 4995 l 14460 5055 l 14612 5055 l 14612 5055 l 14492 5025 l 14612 4995 l cp
|
||||
eoclip
|
||||
n 14475 5025 m
|
||||
14925 5025 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 14612 4995 m 14492 5025 l 14612 5055 l col0 s
|
||||
% arrowhead
|
||||
n 14788 5055 m 14908 5025 l 14788 4995 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
15688 5055 m 15840 5055 l 15840 4995 l 15688 4995 l 15688 4995 l 15808 5025 l 15688 5055 l cp
|
||||
15137 4995 m 14985 4995 l 14985 5055 l 15137 5055 l 15137 5055 l 15017 5025 l 15137 4995 l cp
|
||||
eoclip
|
||||
n 15000 5025 m
|
||||
15825 5025 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 15137 4995 m 15017 5025 l 15137 5055 l col0 s
|
||||
% arrowhead
|
||||
n 15688 5055 m 15808 5025 l 15688 4995 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
14638 5355 m 14790 5355 l 14790 5295 l 14638 5295 l 14638 5295 l 14758 5325 l 14638 5355 l cp
|
||||
14612 5295 m 14460 5295 l 14460 5355 l 14612 5355 l 14612 5355 l 14492 5325 l 14612 5295 l cp
|
||||
eoclip
|
||||
n 14475 5325 m 14550 5325 l 14625 5325 l 14700 5325 l 14775 5325 l 14700 5325 l
|
||||
|
||||
14775 5325 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 14612 5295 m 14492 5325 l 14612 5355 l col0 s
|
||||
% arrowhead
|
||||
n 14638 5355 m 14758 5325 l 14638 5295 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
15163 5355 m 15315 5355 l 15315 5295 l 15163 5295 l 15163 5295 l 15283 5325 l 15163 5355 l cp
|
||||
15137 5295 m 14985 5295 l 14985 5355 l 15137 5355 l 15137 5355 l 15017 5325 l 15137 5295 l cp
|
||||
eoclip
|
||||
n 15000 5325 m 15075 5325 l 15150 5325 l 15225 5325 l
|
||||
15300 5325 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 15137 5295 m 15017 5325 l 15137 5355 l col0 s
|
||||
% arrowhead
|
||||
n 15163 5355 m 15283 5325 l 15163 5295 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
15688 5355 m 15840 5355 l 15840 5295 l 15688 5295 l 15688 5295 l 15808 5325 l 15688 5355 l cp
|
||||
15587 5295 m 15435 5295 l 15435 5355 l 15587 5355 l 15587 5355 l 15467 5325 l 15587 5295 l cp
|
||||
eoclip
|
||||
n 15450 5325 m 15525 5325 l 15600 5325 l 15675 5325 l 15750 5325 l
|
||||
15825 5325 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 15587 5295 m 15467 5325 l 15587 5355 l col0 s
|
||||
% arrowhead
|
||||
n 15688 5355 m 15808 5325 l 15688 5295 l col0 s
|
||||
% Polyline
|
||||
[60] 0 sd
|
||||
n 14475 5025 m
|
||||
15825 5025 l gs col0 s gr [] 0 sd
|
||||
% Polyline
|
||||
[60] 0 sd
|
||||
n 14475 5325 m
|
||||
15825 5325 l gs col0 s gr [] 0 sd
|
||||
% Polyline
|
||||
[60] 0 sd
|
||||
n 14475 5550 m
|
||||
15825 5550 l gs col0 s gr [] 0 sd
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 4575 m
|
||||
gs 1 -1 sc (epoch=13) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 4830 m
|
||||
gs 1 -1 sc (files:) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 5085 m
|
||||
gs 1 -1 sc ( "foo.seq_a.006") col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 5340 m
|
||||
gs 1 -1 sc ( "foo.seq_b.007") col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 5595 m
|
||||
gs 1 -1 sc ( "foo.seq_b.008") col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 4275 m
|
||||
gs 1 -1 sc (FLU \(proc\)) col0 sh gr
|
||||
% Polyline
|
||||
15.000 slw
|
||||
n 12750 4050 m 15975 4050 l 15975 5775 l 12750 5775 l
|
||||
cp gs col0 s gr
|
||||
% Polyline
|
||||
n 12750 2775 m 15150 2775 l 15150 3900 l 12750 3900 l
|
||||
cp gs col0 s gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 3000 m
|
||||
gs 1 -1 sc (Sequencer \(proc\)) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 3300 m
|
||||
gs 1 -1 sc (epoch=13) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
12825 3555 m
|
||||
gs 1 -1 sc (map=[{"foo", next_file=8,) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
13500 3750 m
|
||||
gs 1 -1 sc (next_offset=0}...]) col0 sh gr
|
||||
% Polyline
|
||||
n 5700 3975 m 5700 4275 l 8250 4275 l 8250 3075 l 7950 3075 l 7950 3975 l
|
||||
|
||||
5700 3975 l cp gs col0 s gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5775 4200 m
|
||||
gs 1 -1 sc (server logic) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5775 3060 m
|
||||
gs 1 -1 sc (Append <<123 bytes>>) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5775 3315 m
|
||||
gs 1 -1 sc (to a file with prefix) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5775 3570 m
|
||||
gs 1 -1 sc ("foo".) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
5775 2700 m
|
||||
gs 1 -1 sc (CLIENT \(proc\)) col0 sh gr
|
||||
% Polyline
|
||||
gs clippath
|
||||
5970 3763 m 5970 3915 l 6030 3915 l 6030 3763 l 6030 3763 l 6000 3883 l 5970 3763 l cp
|
||||
eoclip
|
||||
n 6000 3600 m
|
||||
6000 3900 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 5970 3763 m 6000 3883 l 6030 3763 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
6630 3737 m 6630 3585 l 6570 3585 l 6570 3737 l 6570 3737 l 6600 3617 l 6630 3737 l cp
|
||||
eoclip
|
||||
n 6600 3900 m 6600 3825 l 6600 3750 l 6600 3675 l
|
||||
6600 3600 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 6630 3737 m 6600 3617 l 6570 3737 l col0 s
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
6675 3900 m
|
||||
gs 1 -1 sc (ok) col0 sh gr
|
||||
% Polyline
|
||||
15.000 slw
|
||||
n 5550 4350 m 8325 4350 l 8325 2475 l 5550 2475 l
|
||||
cp gs col0 s gr
|
||||
% Polyline
|
||||
gs clippath
|
||||
8143 4500 m 8035 4393 l 7993 4435 l 8100 4543 l 8100 4543 l 8037 4437 l 8143 4500 l cp
|
||||
eoclip
|
||||
n 12525 5175 m 8775 5175 l
|
||||
8025 4425 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 8143 4500 m 8037 4437 l 8100 4543 l col0 s
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
11625 5100 m
|
||||
gs 1 -1 sc (ok) col0 sh gr
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
5970 4663 m 5970 4815 l 6030 4815 l 6030 4663 l 6030 4663 l 6000 4783 l 5970 4663 l cp
|
||||
eoclip
|
||||
n 6000 4425 m
|
||||
6000 4800 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 5970 4663 m 6000 4783 l 6030 4663 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
6630 4562 m 6630 4410 l 6570 4410 l 6570 4562 l 6570 4562 l 6600 4442 l 6630 4562 l cp
|
||||
eoclip
|
||||
n 6600 4800 m 6600 4650 l 6600 4575 l 6600 4500 l
|
||||
6600 4425 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 6630 4562 m 6600 4442 l 6570 4562 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
12388 2730 m 12540 2730 l 12540 2670 l 12388 2670 l 12388 2670 l 12508 2700 l 12388 2730 l cp
|
||||
eoclip
|
||||
n 8475 2700 m
|
||||
12525 2700 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 12388 2730 m 12508 2700 l 12388 2670 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
8612 2970 m 8460 2970 l 8460 3030 l 8612 3030 l 8612 3030 l 8492 3000 l 8612 2970 l cp
|
||||
eoclip
|
||||
n 12525 3000 m
|
||||
8475 3000 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 8612 2970 m 8492 3000 l 8612 3030 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
8612 3645 m 8460 3645 l 8460 3705 l 8612 3705 l 8612 3705 l 8492 3675 l 8612 3645 l cp
|
||||
eoclip
|
||||
n 12525 6900 m 12000 6900 l 12000 3675 l
|
||||
8475 3675 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 8612 3645 m 8492 3675 l 8612 3705 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
12388 3330 m 12540 3330 l 12540 3270 l 12388 3270 l 12388 3270 l 12508 3300 l 12388 3330 l cp
|
||||
eoclip
|
||||
n 8475 3975 m 12300 3975 l 12300 3300 l
|
||||
12525 3300 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 12388 3330 m 12508 3300 l 12388 3270 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
12388 4905 m 12540 4905 l 12540 4845 l 12388 4845 l 12388 4845 l 12508 4875 l 12388 4905 l cp
|
||||
eoclip
|
||||
n 8250 4425 m 8700 4875 l
|
||||
12525 4875 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 12388 4905 m 12508 4875 l 12388 4845 l col0 s
|
||||
% Polyline
|
||||
15.000 slw
|
||||
n 12675 2400 m 16050 2400 l 16050 7875 l 12675 7875 l
|
||||
cp gs col0 s gr
|
||||
% Polyline
|
||||
gs clippath
|
||||
8612 4245 m 8460 4245 l 8460 4305 l 8612 4305 l 8612 4305 l 8492 4275 l 8612 4245 l cp
|
||||
eoclip
|
||||
n 12525 3600 m 12375 3600 l 12375 4275 l
|
||||
8475 4275 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 8612 4245 m 8492 4275 l 8612 4305 l col0 s
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8850 5625 m
|
||||
gs 1 -1 sc (Write to FLU B -> ok) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8850 6135 m
|
||||
gs 1 -1 sc (Write to FLU C -> ok) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8550 2625 m
|
||||
gs 1 -1 sc (Request 123 bytes, prefix="foo", epoch=12) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
11100 2925 m
|
||||
gs 1 -1 sc ({bad_epoch,13}) col4 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
10875 3600 m
|
||||
gs 1 -1 sc ({ok, proj=...}) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8550 4500 m
|
||||
gs 1 -1 sc (Write <<123 bytes>> to) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8550 4755 m
|
||||
gs 1 -1 sc (file="foo.seq_a.008", offset=0) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
13575 2625 m
|
||||
gs 1 -1 sc (Server A) col0 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
8550 3900 m
|
||||
gs 1 -1 sc (Req. 123 bytes, prefix="foo", epoch=13) col0 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
6075 3825 m
|
||||
gs 1 -1 sc (1) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
6075 4650 m
|
||||
gs 1 -1 sc (2) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
6675 4650 m
|
||||
gs 1 -1 sc (3) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
10950 2850 m
|
||||
gs 1 -1 sc (5) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
10725 3525 m
|
||||
gs 1 -1 sc (7) col16 sh gr
|
||||
/Times-Bold-iso ff 200.00 scf sf
|
||||
9375 4200 m
|
||||
gs 1 -1 sc (file="foo.seq_a.008", offset=0) col0 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8400 3225 m
|
||||
gs 1 -1 sc (6) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8400 2625 m
|
||||
gs 1 -1 sc (4) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
6675 3750 m
|
||||
gs 1 -1 sc (16) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8400 3900 m
|
||||
gs 1 -1 sc (8) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
9225 4200 m
|
||||
gs 1 -1 sc (9) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8400 4500 m
|
||||
gs 1 -1 sc (10) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8475 5625 m
|
||||
gs 1 -1 sc (12,13) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
8475 6075 m
|
||||
gs 1 -1 sc (14,15) col16 sh gr
|
||||
/Times-Roman-iso ff 166.67 scf sf
|
||||
11400 5100 m
|
||||
gs 1 -1 sc (11) col16 sh gr
|
||||
% Polyline
|
||||
15.000 slw
|
||||
gs clippath
|
||||
12388 6630 m 12540 6630 l 12540 6570 l 12388 6570 l 12388 6570 l 12508 6600 l 12388 6630 l cp
|
||||
eoclip
|
||||
n 8475 3300 m 12150 3300 l 12150 6600 l
|
||||
12525 6600 l gs col0 s gr gr
|
||||
|
||||
% arrowhead
|
||||
7.500 slw
|
||||
n 12388 6630 m 12508 6600 l 12388 6570 l col0 s
|
||||
% here ends figure;
|
||||
pagefooter
|
||||
showpage
|
||||
%%Trailer
|
||||
%EOF
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,145 +0,0 @@
|
|||
%!PS-Adobe-3.0 EPSF-2.0
|
||||
%%BoundingBox: 0 0 420.000000 166.599991
|
||||
%%Creator: mscgen 0.18
|
||||
%%EndComments
|
||||
0.700000 0.700000 scale
|
||||
0 0 moveto
|
||||
0 238 lineto
|
||||
600 238 lineto
|
||||
600 0 lineto
|
||||
closepath
|
||||
clip
|
||||
%PageTrailer
|
||||
%Page: 1 1
|
||||
/Helvetica findfont
|
||||
10 scalefont
|
||||
setfont
|
||||
/Helvetica findfont
|
||||
12 scalefont
|
||||
setfont
|
||||
0 238 translate
|
||||
/mtrx matrix def
|
||||
/ellipse
|
||||
{ /endangle exch def
|
||||
/startangle exch def
|
||||
/ydia exch def
|
||||
/xdia exch def
|
||||
/y exch def
|
||||
/x exch def
|
||||
/savematrix mtrx currentmatrix def
|
||||
x y translate
|
||||
xdia 2 div ydia 2 div scale
|
||||
1 -1 scale
|
||||
0 0 1 startangle endangle arc
|
||||
savematrix setmatrix
|
||||
} def
|
||||
(client) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 75 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
75 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(Projection) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 225 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
225 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(ProjStore_C) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 375 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
375 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
(FLU_C) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup dup newpath 525 -17 moveto 2 div neg 0 rmoveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
525 -15 moveto dup stringwidth pop 2 div neg 0 rmoveto show
|
||||
newpath 75 -22 moveto 75 -49 lineto stroke
|
||||
newpath 225 -22 moveto 225 -49 lineto stroke
|
||||
newpath 375 -22 moveto 375 -49 lineto stroke
|
||||
newpath 525 -22 moveto 525 -49 lineto stroke
|
||||
newpath 75 -35 moveto 225 -35 lineto stroke
|
||||
newpath 225 -35 moveto 215 -41 lineto stroke
|
||||
(get current) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 122 -33 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
122 -33 moveto show
|
||||
newpath 75 -49 moveto 75 -76 lineto stroke
|
||||
newpath 225 -49 moveto 225 -76 lineto stroke
|
||||
newpath 375 -49 moveto 375 -76 lineto stroke
|
||||
newpath 525 -49 moveto 525 -76 lineto stroke
|
||||
newpath 225 -62 moveto 75 -62 lineto stroke
|
||||
newpath 75 -62 moveto 85 -68 lineto stroke
|
||||
(ok, #12...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 126 -60 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
126 -60 moveto show
|
||||
newpath 75 -76 moveto 75 -103 lineto stroke
|
||||
newpath 225 -76 moveto 225 -103 lineto stroke
|
||||
newpath 375 -76 moveto 375 -103 lineto stroke
|
||||
newpath 525 -76 moveto 525 -103 lineto stroke
|
||||
newpath 75 -89 moveto 525 -89 lineto stroke
|
||||
newpath 525 -89 moveto 515 -95 lineto stroke
|
||||
(read "foo.seq_a.009" offset=447 bytes=123 epoch=12) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 157 -87 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
157 -87 moveto show
|
||||
newpath 75 -103 moveto 75 -130 lineto stroke
|
||||
newpath 225 -103 moveto 225 -130 lineto stroke
|
||||
newpath 375 -103 moveto 375 -130 lineto stroke
|
||||
newpath 525 -103 moveto 525 -130 lineto stroke
|
||||
newpath 525 -116 moveto 75 -116 lineto stroke
|
||||
newpath 75 -116 moveto 85 -122 lineto stroke
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
(bad_epoch, 13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 261 -114 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
1.000000 0.000000 0.000000 setrgbcolor
|
||||
261 -114 moveto show
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
newpath 75 -130 moveto 75 -157 lineto stroke
|
||||
newpath 225 -130 moveto 225 -157 lineto stroke
|
||||
newpath 375 -130 moveto 375 -157 lineto stroke
|
||||
newpath 525 -130 moveto 525 -157 lineto stroke
|
||||
newpath 75 -143 moveto 375 -143 lineto stroke
|
||||
newpath 375 -143 moveto 365 -149 lineto stroke
|
||||
(get epoch #13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 187 -141 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
187 -141 moveto show
|
||||
newpath 75 -157 moveto 75 -184 lineto stroke
|
||||
newpath 225 -157 moveto 225 -184 lineto stroke
|
||||
newpath 375 -157 moveto 375 -184 lineto stroke
|
||||
newpath 525 -157 moveto 525 -184 lineto stroke
|
||||
newpath 375 -170 moveto 75 -170 lineto stroke
|
||||
newpath 75 -170 moveto 85 -176 lineto stroke
|
||||
(ok, #13...) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 201 -168 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
201 -168 moveto show
|
||||
newpath 75 -184 moveto 75 -211 lineto stroke
|
||||
newpath 225 -184 moveto 225 -211 lineto stroke
|
||||
newpath 375 -184 moveto 375 -211 lineto stroke
|
||||
newpath 525 -184 moveto 525 -211 lineto stroke
|
||||
newpath 75 -197 moveto 525 -197 lineto stroke
|
||||
newpath 525 -197 moveto 515 -203 lineto stroke
|
||||
(read "foo.seq_a.009" offset=447 bytes=123 epoch=13) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 157 -195 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
157 -195 moveto show
|
||||
newpath 75 -211 moveto 75 -238 lineto stroke
|
||||
newpath 225 -211 moveto 225 -238 lineto stroke
|
||||
newpath 375 -211 moveto 375 -238 lineto stroke
|
||||
newpath 525 -211 moveto 525 -238 lineto stroke
|
||||
newpath 525 -224 moveto 75 -224 lineto stroke
|
||||
newpath 75 -224 moveto 85 -230 lineto stroke
|
||||
(ok, <<...123...>>) dup stringwidth
|
||||
1.000000 1.000000 1.000000 setrgbcolor
|
||||
pop dup newpath 257 -222 moveto 0 rlineto 0 11 rlineto neg 0 rlineto closepath fill
|
||||
0.000000 0.000000 0.000000 setrgbcolor
|
||||
257 -222 moveto show
|
||||
File diff suppressed because it is too large
Load diff
2
ebin/.gitignore
vendored
2
ebin/.gitignore
vendored
|
|
@ -1,2 +0,0 @@
|
|||
*.beam
|
||||
*.app
|
||||
1
edoc/.gitignore
vendored
Normal file
1
edoc/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
tmp.*
|
||||
13
edoc/edoc-info
Normal file
13
edoc/edoc-info
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
%% encoding: UTF-8
|
||||
{application,machi}.
|
||||
{packages,[]}.
|
||||
{modules,[machi_admin_util,machi_app,machi_basho_bench_driver,
|
||||
machi_chain_manager1,machi_chain_repair,machi_chash,machi_cinfo,
|
||||
machi_config,machi_cr_client,machi_csum_table,machi_dt,
|
||||
machi_file_proxy,machi_file_proxy_sup,machi_fitness,machi_flu1,
|
||||
machi_flu1_client,machi_flu_filename_mgr,machi_flu_metadata_mgr,
|
||||
machi_flu_metadata_mgr_sup,machi_flu_psup,machi_flu_sup,
|
||||
machi_lifecycle_mgr,machi_merkle_tree,machi_pb_high_client,
|
||||
machi_pb_translate,machi_plist,machi_projection,
|
||||
machi_projection_store,machi_proxy_flu1_client,machi_sup,machi_util,
|
||||
machi_yessir_client]}.
|
||||
BIN
edoc/erlang.png
Normal file
BIN
edoc/erlang.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 2.1 KiB |
17
edoc/index.html
Normal file
17
edoc/index.html
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<title>The machi application</title>
|
||||
</head>
|
||||
<frameset cols="20%,80%">
|
||||
<frame src="modules-frame.html" name="modulesFrame" title="">
|
||||
|
||||
<frame src="overview-summary.html" name="overviewFrame" title="">
|
||||
<noframes>
|
||||
<h2>This page uses frames</h2>
|
||||
<p>Your browser does not accept frames.
|
||||
<br>You should go to the <a href="overview-summary.html">non-frame version</a> instead.
|
||||
</p>
|
||||
</noframes>
|
||||
</frameset>
|
||||
</html>
|
||||
50
edoc/machi_admin_util.html
Normal file
50
edoc/machi_admin_util.html
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_admin_util</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_admin_util</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Machi chain replication administration utilities.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>Machi chain replication administration utilities.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#verify_file_checksums_local-3">verify_file_checksums_local/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#verify_file_checksums_local-4">verify_file_checksums_local/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#verify_file_checksums_remote-3">verify_file_checksums_remote/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#verify_file_checksums_remote-4">verify_file_checksums_remote/4</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="verify_file_checksums_local-3">verify_file_checksums_local/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verify_file_checksums_local(Sock1::port(), EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Path::binary() | list()) -> {ok, [tuple()]} | {error, term()}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="verify_file_checksums_local-4">verify_file_checksums_local/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verify_file_checksums_local(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Path::binary() | list()) -> {ok, [tuple()]} | {error, term()}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="verify_file_checksums_remote-3">verify_file_checksums_remote/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verify_file_checksums_remote(Sock1::port(), EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::binary() | list()) -> {ok, [tuple()]} | {error, term()}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="verify_file_checksums_remote-4">verify_file_checksums_remote/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verify_file_checksums_remote(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::binary() | list()) -> {ok, [tuple()]} | {error, term()}</tt><br></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
42
edoc/machi_app.html
Normal file
42
edoc/machi_app.html
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_app</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_app</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Start the top-level supervisor for the Machi application.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="application.html"><tt>application</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Start the top-level supervisor for the Machi application.</p>
|
||||
|
||||
See <a href="machi_flu_psup.html"><code>machi_flu_psup</code></a> for an illustration of the entire Machi
|
||||
application process structure.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#start-2">start/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop-1">stop/1</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="start-2">start/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start(StartType, StartArgs) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop-1">stop/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop(State) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
64
edoc/machi_basho_bench_driver.html
Normal file
64
edoc/machi_basho_bench_driver.html
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_basho_bench_driver</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_basho_bench_driver</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>A simple basho_bench driver for Machi.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>A simple basho_bench driver for Machi</p>
|
||||
|
||||
<p>Basho_bench was originally developed to stress test key-value
|
||||
stores (as was YCSB and several other mechmarking tools). A person
|
||||
can consider the UNIX file system to be a key-value store and thus
|
||||
use basho_bench to measure its performance under a certain
|
||||
workload. Machi is a bit different than most KV stores in that the
|
||||
client has no direct control over the keys -- Machi servers always
|
||||
assign the keys. The schemes typically used by basho_bench & YCSB
|
||||
to use/mimic key naming conventions used internally ... are
|
||||
difficult to adapt to Machi.</p>
|
||||
|
||||
<p>So, we'll try to manage key reading by using a common ETS table
|
||||
that is populated with:</p>
|
||||
|
||||
<p>1. Key: <code>non_neg_integer()`
|
||||
2. Value: The `{File,Offset,Size}` for a chunk previously written.
|
||||
|
||||
At startup time, basho_bench can use the `list_files</code> and
|
||||
<code>checksum_list</code> API operations to fetch all of the
|
||||
<code>{File,Offset,Size}` tuples that currently reside in the cluster.
|
||||
Also, optionally (?), each new `append</code> operation by the b_b driver
|
||||
could add new entries to this ETS table.</p>
|
||||
|
||||
Now we can use various integer-centric key generators that are
|
||||
already bundled with basho_bench.
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#new-1">new/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#run-4">run/4</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="new-1">new/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>new(Id) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="run-4">run/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>run(X1, KeyGen, ValueGen, M) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
289
edoc/machi_chain_manager1.html
Normal file
289
edoc/machi_chain_manager1.html
Normal file
|
|
@ -0,0 +1,289 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_chain_manager1</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_chain_manager1</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The Machi chain manager, Guardian of all things related to
|
||||
Chain Replication state, status, and data replica safety.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>The Machi chain manager, Guardian of all things related to
|
||||
Chain Replication state, status, and data replica safety.</p>
|
||||
|
||||
<p>The Chain Manager is responsible for managing the state of Machi's
|
||||
"Chain Replication" state. This role is roughly analogous to the
|
||||
"Riak Core" application inside of Riak, which takes care of
|
||||
coordinating replica placement and replica repair.</p>
|
||||
|
||||
<p>For each primitive data server in the cluster, a Machi FLU, there
|
||||
is a Chain Manager process that manages its FLU's role within the
|
||||
Machi cluster's Chain Replication scheme. Each Chain Manager
|
||||
process executes locally and independently to manage the
|
||||
distributed state of a single Machi Chain Replication chain.</p>
|
||||
|
||||
Machi's Chain Manager process performs similar tasks as Riak Core's
|
||||
claimant. However, Machi has several active Chain Manager
|
||||
processes, one per FLU server, instead of a single active process
|
||||
like Core's claimant. Each Chain Manager process acts
|
||||
independently; each is constrained so that it will reach consensus
|
||||
via independent computation & action.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#chain_state_transition_is_sane-6">chain_state_transition_is_sane/6</a></td><td>Check if a projection transition is sane & safe: we assume
|
||||
that the caller has checked basic projection data structure contents.</td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#format_status-2">format_status/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_ps-2">get_ps/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_remember_down_list-0">get_remember_down_list/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#has_not_sane-2">has_not_sane/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init_remember_down_list-0">init_remember_down_list/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_chmgr_regname-1">make_chmgr_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#mk-3">mk/3</a></td><td>Create a 2-tuple that describes how <code>UPI1</code> + <code>Repair1</code> are
|
||||
transformed into <code>UPI2</code> in a chain state change.</td></tr>
|
||||
<tr><td valign="top"><a href="#perhaps_call-5">perhaps_call/5</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#ping-1">ping/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#projection_transitions_are_sane-2">projection_transitions_are_sane/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#scan_dir-4">scan_dir/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#set_active-2">set_active/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#set_chain_members-2">set_chain_members/2</a></td><td>Set chain members list.</td></tr>
|
||||
<tr><td valign="top"><a href="#set_chain_members-6">set_chain_members/6</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#simple_chain_state_transition_is_sane-3">simple_chain_state_transition_is_sane/3</a></td><td>A simple technique for checking chain state transition safety.</td></tr>
|
||||
<tr><td valign="top"><a href="#simple_chain_state_transition_is_sane-5">simple_chain_state_transition_is_sane/5</a></td><td>Simple check if a projection transition is sane & safe: we assume
|
||||
that the caller has checked basic projection data structure contents.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-2">start_link/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-3">start_link/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop-1">stop/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#strip_dbg2-1">strip_dbg2/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#test_calc_projection-2">test_calc_projection/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#test_read_latest_public_projection-2">test_read_latest_public_projection/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#test_write_public_projection-2">test_write_public_projection/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trigger_react_to_env-1">trigger_react_to_env/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#update_remember_down_list-1">update_remember_down_list/1</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="chain_state_transition_is_sane-6">chain_state_transition_is_sane/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>chain_state_transition_is_sane(Author1, UPI1, Repair1, Author2, UPI2, RelativeToServer) -> any()</tt></p>
|
||||
</div><p><p>Check if a projection transition is sane & safe: we assume
|
||||
that the caller has checked basic projection data structure contents.</p>
|
||||
|
||||
NOTE: The return value convention is <code>true</code> for sane/safe and <code>term() /=
|
||||
true</code> for any unsafe/insane value. This function (and its callee
|
||||
functions) are the only functions (throughout all of the chain state
|
||||
transition sanity checking functions) that is allowed to return <code>false</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="format_status-2">format_status/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>format_status(Opt, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_ps-2">get_ps/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_ps(Projection_v1, Acc) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_remember_down_list-0">get_remember_down_list/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_remember_down_list() -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Call, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Cast, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Msg, Ch_mgr) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="has_not_sane-2">has_not_sane/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>has_not_sane(Projection_v1, Acc) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init_remember_down_list-0">init_remember_down_list/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init_remember_down_list() -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_chmgr_regname-1">make_chmgr_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_chmgr_regname(A) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="mk-3">mk/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>mk(UPI1, Repair1, UPI2) -> any()</tt></p>
|
||||
</div><p><p>Create a 2-tuple that describes how <code>UPI1</code> + <code>Repair1</code> are
|
||||
transformed into <code>UPI2</code> in a chain state change.</p>
|
||||
|
||||
<p>The 1st part of the 2-tuple is a list of <code>keep</code> and <code>del</code> instructions,
|
||||
relative to the items in UPI1 and whether they are present (<code>keep</code>) or
|
||||
absent (<code>del</code>) in <code>UPI2</code>.</p>
|
||||
|
||||
The 2nd part of the 2-tuple is <code>list(non_neg_integer()|error)</code> that
|
||||
describes the relative order of items in <code>Repair1</code> that appear in
|
||||
<code>UPI2</code>. The <code>error</code> atom is used to denote items not present in
|
||||
<code>Repair1</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="perhaps_call-5">perhaps_call/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>perhaps_call(ProxyPid, MyName, Partitions, FLU, DoIt) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="ping-1">ping/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>ping(Pid) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="projection_transitions_are_sane-2">projection_transitions_are_sane/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>projection_transitions_are_sane(Ps, RelativeToServer) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="scan_dir-4">scan_dir/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>scan_dir(Dir, FileFilterFun, FoldEachFun, FoldEachAcc) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="set_active-2">set_active/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>set_active(Pid, Boolean) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="set_chain_members-2">set_chain_members/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>set_chain_members(Pid, MembersDict) -> any()</tt></p>
|
||||
</div><p><p>Set chain members list.</p>
|
||||
|
||||
NOTE: This implementation is a bit brittle, in that an author with
|
||||
higher rank may try to re-suggest the old membership list if it
|
||||
races with an author of lower rank. For now, we suggest calling
|
||||
set_chain_members() first on the author of highest rank and finish
|
||||
with lowest rank, i.e. name z* first, name a* last.</p>
|
||||
|
||||
<h3 class="function"><a name="set_chain_members-6">set_chain_members/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>set_chain_members(Pid, ChainName, OldEpoch, CMode, MembersDict, Witness_list) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="simple_chain_state_transition_is_sane-3">simple_chain_state_transition_is_sane/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>simple_chain_state_transition_is_sane(UPI1, Repair1, UPI2) -> any()</tt></p>
|
||||
</div><p><p>A simple technique for checking chain state transition safety.</p>
|
||||
|
||||
<p>Math tells us that any change state <code>UPI1</code> plus <code>Repair1</code> to state
|
||||
<code>UPI2</code> is OK as long as <code>UPI2</code> is a concatenation of some
|
||||
order-preserving combination from <code>UPI1</code> with some order-preserving
|
||||
combination from <code>Repair1</code>.</p>
|
||||
|
||||
<pre> Good_UPI2s = [ X ++ Y || X <- machi_util:ordered_combinations(UPI1),
|
||||
Y <- machi_util:ordered_combinations(Repair1)]</pre>
|
||||
|
||||
<p>Rather than creating that list and then checking if <code>UPI2</code> is in
|
||||
it, we try a <code>diff</code>-like technique to check for basic state
|
||||
transition safety. See docs for <a href="#mk-3"><code>mk/3</code></a> for more detail.</p>
|
||||
|
||||
<pre> 2> machi_chain_manager1:mk([a,b], [], [a]).
|
||||
{[keep,del],[]} %% good transition
|
||||
3> machi_chain_manager1:mk([a,b], [], [b,a]).
|
||||
{[del,keep],[]} %% bad transition: too few 'keep' for UPI2's length 2
|
||||
4> machi_chain_manager1:mk([a,b], [c,d,e], [a,d]).
|
||||
{[keep,del],[2]} %% good transition
|
||||
5> machi_chain_manager1:mk([a,b], [c,d,e], [a,bogus]).
|
||||
{[keep,del],[error]} %% bad transition: 'bogus' not in Repair1</pre></p>
|
||||
|
||||
<h3 class="function"><a name="simple_chain_state_transition_is_sane-5">simple_chain_state_transition_is_sane/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>simple_chain_state_transition_is_sane(Author1, UPI1, Repair1, Author2, UPI2) -> any()</tt></p>
|
||||
</div><p><p>Simple check if a projection transition is sane & safe: we assume
|
||||
that the caller has checked basic projection data structure contents.</p>
|
||||
|
||||
NOTE: The return value convention is <code>true</code> for sane/safe and
|
||||
<code>term() /= true</code> for any unsafe/insane value.</p>
|
||||
|
||||
<h3 class="function"><a name="start_link-2">start_link/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(MyName, MembersDict) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-3">start_link/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(MyName, MembersDict, MgrOpts) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop-1">stop/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop(Pid) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="strip_dbg2-1">strip_dbg2/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>strip_dbg2(P) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="test_calc_projection-2">test_calc_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>test_calc_projection(Pid, KeepRunenvP) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="test_read_latest_public_projection-2">test_read_latest_public_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>test_read_latest_public_projection(Pid, ReadRepairP) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="test_write_public_projection-2">test_write_public_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>test_write_public_projection(Pid, Proj) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trigger_react_to_env-1">trigger_react_to_env/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trigger_react_to_env(Pid) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="update_remember_down_list-1">update_remember_down_list/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>update_remember_down_list(FLU) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
91
edoc/machi_chain_repair.html
Normal file
91
edoc/machi_chain_repair.html
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_chain_repair</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_chain_repair</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Perform "chain repair", i.e., resynchronization of Machi file
|
||||
contents and metadata as servers are (re-)added to the chain.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Perform "chain repair", i.e., resynchronization of Machi file
|
||||
contents and metadata as servers are (re-)added to the chain.</p>
|
||||
|
||||
<p>The implementation here is a very basic one, and is probably a bit
|
||||
slower than the original "demo day" implementation at
|
||||
<a href="https://github.com/basho/machi/blob/master/prototype/demo-day-hack/file0_repair_server.escript" target="_top"><tt>https://github.com/basho/machi/blob/master/prototype/demo-day-hack/file0_repair_server.escript</tt></a></p>
|
||||
|
||||
<p>It's so easy to bikeshed this into a 1 year programming exercise.</p>
|
||||
|
||||
<p>General TODO note: There are a lot of areas for exploiting parallelism here.
|
||||
I've set the bikeshed aside for now, but "make repair faster" has a
|
||||
lot of room for exploiting concurrency, overlapping reads & writes,
|
||||
etc etc. There are also lots of different trade-offs to make with
|
||||
regard to RAM use vs. disk use.</p>
|
||||
|
||||
<p>There's no reason why repair can't be done:</p>
|
||||
|
||||
<ol>
|
||||
<li> Repair in parallel across multiple repairees ... Optimization.
|
||||
</li>
|
||||
<li> Repair multiple byte ranges concurrently ... Optimization.
|
||||
</li>
|
||||
<li><p> Use bigger chunks than the client originally used to write the file
|
||||
... Optimization ... but it would be the easiest to implement, e.g. use
|
||||
constant-sized 4MB chunks. Unfortuntely, it would also destroy
|
||||
the ability to verify here that the chunk checksums are correct
|
||||
*and* also propagate the correct checksum metadata to the
|
||||
destination FLU.</p>
|
||||
|
||||
As an additional optimization, add a bit of #2 to start the next
|
||||
read while the current write is still in progress.
|
||||
</li>
|
||||
<li> The current method centralizes the "smarts" required to compare
|
||||
checksum differences ... move some computation to each FLU, then use
|
||||
a Merkle- or other-compression-style scheme to reduce the amount of
|
||||
data sent across a network.
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<p>Most/all of this could be executed in parallel on each FLU relative to
|
||||
its own files. Then, in another TODO option, perhaps build a Merkle tree
|
||||
or other summary of the local files and send that data structure to the
|
||||
repair coordinator.</p>
|
||||
|
||||
<p>Also, as another TODO note, repair_both_present() in the
|
||||
prototype/demo-day code uses an optimization of calculating the MD5
|
||||
checksum of the chunk checksum data as it arrives, and if the two MD5s
|
||||
match, then we consider the two files in sync. If there isn't a match,
|
||||
then we sort the lines and try another MD5, and if they match, then we're
|
||||
in sync. In theory, that's lower overhead than the procedure used here.</p>
|
||||
|
||||
NOTE that one reason I chose the "directives list" method is to have an
|
||||
option, later, of choosing to repair a subset of repairee FLUs if there
|
||||
is a big discrepency between out of sync files: e.g., if FLU x has N
|
||||
bytes out of sync but FLU y has 50N bytes out of sync, then it's likely
|
||||
better to repair x only so that x can return to the UPI list quickly.
|
||||
Also, in the event that all repairees are roughly comparably out of sync,
|
||||
then the repair network traffic can be minimized by reading each chunk
|
||||
only once.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#repair-7">repair/7</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="repair-7">repair/7</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>repair(ConsistencyMode, Src, Repairing, UPI, MembersDict, ETS, Opts) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
171
edoc/machi_chash.html
Normal file
171
edoc/machi_chash.html
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_chash</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_chash</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#types">Data Types</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Consistent hashing library.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Consistent hashing library. Also known as "random slicing".</p>
|
||||
|
||||
This code was originally from the Hibari DB source code at
|
||||
<a href="https://github.com/hibari" target="_top"><tt>https://github.com/hibari</tt></a>
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-float_map">float_map()</a></h3>
|
||||
<p><tt>float_map() = [{<a href="#type-owner_name">owner_name()</a>, float()}]</tt></p>
|
||||
<p> A float map subdivides the unit interval, starting at 0.0, to
|
||||
partitions that are assigned to various owners. The sum of all
|
||||
floats must be exactly 1.0 (or close enough for floating point
|
||||
purposes).</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-float_tree">float_tree()</a></h3>
|
||||
<p><b>abstract datatype</b>: <tt>float_tree()</tt></p>
|
||||
<p> We can't use gb_trees:tree() because 'nil' (the empty tree) is
|
||||
never valid in our case. But teaching Dialyzer that is difficult.</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-nextfloat_list">nextfloat_list()</a></h3>
|
||||
<p><tt>nextfloat_list() = [{float(), <a href="#type-brick">brick()</a>}]</tt></p>
|
||||
<p>A nextfloat_list
|
||||
differs from a float_map in two respects: 1) nextfloat_list contains
|
||||
tuples with the brick name in 2nd position, 2) the float() at each
|
||||
position I_n > I_m, for all n, m such that n > m.
|
||||
For example, a nextfloat_list of the float_map example above,
|
||||
[{0.25, {br1, nd1}}, {0.75, {br2, nd1}}, {1.0, {br3, nd1}].</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-owner_int_range">owner_int_range()</a></h3>
|
||||
<p><tt>owner_int_range() = {<a href="#type-owner_name">owner_name()</a>, non_neg_integer(), non_neg_integer()}</tt></p>
|
||||
<p> Used when "prettying" a float map.</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-owner_name">owner_name()</a></h3>
|
||||
<p><tt>owner_name() = term()</tt></p>
|
||||
<p> Owner for a range on the unit interval. We are agnostic about its
|
||||
type.</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-owner_weight">owner_weight()</a></h3>
|
||||
<p><tt>owner_weight() = {<a href="#type-owner_name">owner_name()</a>, <a href="#type-weight">weight()</a>}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-owner_weight_list">owner_weight_list()</a></h3>
|
||||
<p><tt>owner_weight_list() = [<a href="#type-owner_weight">owner_weight()</a>]</tt></p>
|
||||
<p> A owner_weight_list is a definition of brick assignments over the
|
||||
unit interval [0.0, 1.0]. The sum of all floats must be 1.0. For
|
||||
example, [{{br1,nd1}, 0.25}, {{br2,nd1}, 0.5}, {{br3,nd1}, 0.25}].</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-weight">weight()</a></h3>
|
||||
<p><tt>weight() = non_neg_integer()</tt></p>
|
||||
<p> For this library, a weight is an integer which specifies the
|
||||
capacity of a "owner" relative to other owners. For example, if
|
||||
owner A with a weight of 10, and if owner B has a weight of 20,
|
||||
then B will be assigned twice as much of the unit interval as A.</p>
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#hash_binary_via_float_map-2">hash_binary_via_float_map/2</a></td><td>Query a float map with a binary (inefficient).</td></tr>
|
||||
<tr><td valign="top"><a href="#hash_binary_via_float_tree-2">hash_binary_via_float_tree/2</a></td><td>Query a float tree with a binary.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_demo_map1-0">make_demo_map1/0</a></td><td>Create a sample float map.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_demo_map2-0">make_demo_map2/0</a></td><td>Create a sample float map.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_float_map-1">make_float_map/1</a></td><td>Create a float map, based on a basic owner weight list.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_float_map-2">make_float_map/2</a></td><td>Create a float map, based on an older float map and a new weight
|
||||
list.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_tree-1">make_tree/1</a></td><td>Create a float tree, which is the rapid lookup data structure
|
||||
for consistent hash queries.</td></tr>
|
||||
<tr><td valign="top"><a href="#pretty_with_integers-2">pretty_with_integers/2</a></td><td>Make a pretty/human-friendly version of a float map that describes
|
||||
integer ranges between 1 and <code>Scale</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#pretty_with_integers-3">pretty_with_integers/3</a></td><td>Make a pretty/human-friendly version of a float map (based
|
||||
upon a float map created from <code>OldWeights</code> and <code>NewWeights</code>) that
|
||||
describes integer ranges between 1 and <code>Scale</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#query_tree-2">query_tree/2</a></td><td>Low-level function for querying a float tree: the (floating
|
||||
point) point within the unit interval.</td></tr>
|
||||
<tr><td valign="top"><a href="#sum_map_weights-1">sum_map_weights/1</a></td><td>Create a human-friendly summary of a float map.</td></tr>
|
||||
<tr><td valign="top"><a href="#zzz_usage_details-0">zzz_usage_details/0</a></td><td>Various usage examples, see source code below this function
|
||||
for full details.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="hash_binary_via_float_map-2">hash_binary_via_float_map/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>hash_binary_via_float_map(Key::binary(), Map::<a href="#type-float_map">float_map()</a>) -> {float(), <a href="#type-owner_name">owner_name()</a>}</tt><br></p>
|
||||
</div><p>Query a float map with a binary (inefficient).</p>
|
||||
|
||||
<h3 class="function"><a name="hash_binary_via_float_tree-2">hash_binary_via_float_tree/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>hash_binary_via_float_tree(Key::binary(), Tree::<a href="#type-float_tree">float_tree()</a>) -> {float(), <a href="#type-owner_name">owner_name()</a>}</tt><br></p>
|
||||
</div><p>Query a float tree with a binary.</p>
|
||||
|
||||
<h3 class="function"><a name="make_demo_map1-0">make_demo_map1/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_demo_map1() -> <a href="#type-float_map">float_map()</a></tt><br></p>
|
||||
</div><p>Create a sample float map.</p>
|
||||
|
||||
<h3 class="function"><a name="make_demo_map2-0">make_demo_map2/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_demo_map2() -> <a href="#type-float_map">float_map()</a></tt><br></p>
|
||||
</div><p>Create a sample float map.</p>
|
||||
|
||||
<h3 class="function"><a name="make_float_map-1">make_float_map/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_float_map(NewOwnerWeights::<a href="#type-owner_weight_list">owner_weight_list()</a>) -> <a href="#type-float_map">float_map()</a></tt><br></p>
|
||||
</div><p>Create a float map, based on a basic owner weight list.</p>
|
||||
|
||||
<h3 class="function"><a name="make_float_map-2">make_float_map/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_float_map(OldFloatMap::<a href="#type-float_map">float_map()</a>, NewOwnerWeights::<a href="#type-owner_weight_list">owner_weight_list()</a>) -> <a href="#type-float_map">float_map()</a></tt><br></p>
|
||||
</div><p><p>Create a float map, based on an older float map and a new weight
|
||||
list.</p>
|
||||
|
||||
The weights in the new weight list may be different than (or the
|
||||
same as) whatever weights were used to make the older float map.</p>
|
||||
|
||||
<h3 class="function"><a name="make_tree-1">make_tree/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_tree(Map::<a href="#type-float_map">float_map()</a>) -> <a href="#type-float_tree">float_tree()</a></tt><br></p>
|
||||
</div><p>Create a float tree, which is the rapid lookup data structure
|
||||
for consistent hash queries.</p>
|
||||
|
||||
<h3 class="function"><a name="pretty_with_integers-2">pretty_with_integers/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>pretty_with_integers(Map::<a href="#type-float_map">float_map()</a>, Scale::integer()) -> [<a href="#type-owner_int_range">owner_int_range()</a>]</tt><br></p>
|
||||
</div><p>Make a pretty/human-friendly version of a float map that describes
|
||||
integer ranges between 1 and <code>Scale</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="pretty_with_integers-3">pretty_with_integers/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>pretty_with_integers(OldWeights::<a href="#type-owner_weight_list">owner_weight_list()</a>, NewWeights::<a href="#type-owner_weight_list">owner_weight_list()</a>, Scale::integer()) -> [<a href="#type-owner_int_range">owner_int_range()</a>]</tt><br></p>
|
||||
</div><p>Make a pretty/human-friendly version of a float map (based
|
||||
upon a float map created from <code>OldWeights</code> and <code>NewWeights</code>) that
|
||||
describes integer ranges between 1 and <code>Scale</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="query_tree-2">query_tree/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>query_tree(Val::float(), Tree::<a href="#type-float_tree">float_tree()</a>) -> {float(), <a href="#type-owner_name">owner_name()</a>}</tt><br></p>
|
||||
</div><p>Low-level function for querying a float tree: the (floating
|
||||
point) point within the unit interval.</p>
|
||||
|
||||
<h3 class="function"><a name="sum_map_weights-1">sum_map_weights/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>sum_map_weights(Map::<a href="#type-float_map">float_map()</a>) -> {{per_owner, <a href="#type-float_map">float_map()</a>}, {weight_sum, float()}}</tt><br></p>
|
||||
</div><p><p>Create a human-friendly summary of a float map.</p>
|
||||
|
||||
The two parts of the summary are: a per-owner total of the unit
|
||||
interval range(s) owned by each owner, and a total sum of all
|
||||
per-owner ranges (which should be 1.0 but is not enforced).</p>
|
||||
|
||||
<h3 class="function"><a name="zzz_usage_details-0">zzz_usage_details/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>zzz_usage_details() -> any()</tt></p>
|
||||
</div><p>Various usage examples, see source code below this function
|
||||
for full details.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
82
edoc/machi_cinfo.html
Normal file
82
edoc/machi_cinfo.html
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_cinfo</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_cinfo</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>cluster_info callback module for machi specific information
|
||||
gathering.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>cluster_info callback module for machi specific information
|
||||
gathering.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#chain_manager-1">chain_manager/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#cluster_info_generator_funs-0">cluster_info_generator_funs/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#cluster_info_init-0">cluster_info_init/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#dump-0">dump/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#fitness-1">fitness/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#flu1-1">flu1/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#private_projection-1">private_projection/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#public_projection-1">public_projection/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#register-0">register/0</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="chain_manager-1">chain_manager/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>chain_manager(FluName::atom()) -> term()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="cluster_info_generator_funs-0">cluster_info_generator_funs/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>cluster_info_generator_funs() -> [{string(), fun((pid()) -> ok)}]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="cluster_info_init-0">cluster_info_init/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>cluster_info_init() -> ok</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="dump-0">dump/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>dump() -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="fitness-1">fitness/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>fitness(FluName::atom()) -> term()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="flu1-1">flu1/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>flu1(FluName::atom()) -> [{atom(), term()}]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="private_projection-1">private_projection/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>private_projection(FluName::atom()) -> [{atom(), term()}]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="public_projection-1">public_projection/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>public_projection(FluName::atom()) -> [{atom(), term()}]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="register-0">register/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>register() -> ok</tt><br></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
44
edoc/machi_config.html
Normal file
44
edoc/machi_config.html
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_config</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_config</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Configuration consulting utilities.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>Configuration consulting utilities. Some conventions:
|
||||
- The function name should match with exact configuration
|
||||
name in <code>app.config</code> or <code>advanced.config</code> of <code>machi</code> section.
|
||||
- The default value of that configuration is expected to be in
|
||||
cuttlefish schema file. Otherwise some macro in headers may
|
||||
be chosen.
|
||||
- Documentation of the configuration is supposed to be written
|
||||
in cuttlefish schema file, rather than @doc section of the function.
|
||||
- spec of the function should be written.
|
||||
- Returning <code>undefined</code> is strongly discouraged. Return some default
|
||||
value instead.
|
||||
- <code>application:get_env/3</code> is recommended. See <code>max_file_size/0</code> for
|
||||
example.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#max_file_size-0">max_file_size/0</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="max_file_size-0">max_file_size/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>max_file_size() -> pos_integer()</tt><br></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
263
edoc/machi_cr_client.html
Normal file
263
edoc/machi_cr_client.html
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_cr_client</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_cr_client</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Erlang API for the Machi client-implemented Chain Replication
|
||||
(CORFU-style) protocol.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Erlang API for the Machi client-implemented Chain Replication
|
||||
(CORFU-style) protocol.</p>
|
||||
|
||||
<p>See also the docs for <a href="machi_flu1_client.html"><code>machi_flu1_client</code></a> for additional
|
||||
details on data types and operation descriptions.</p>
|
||||
|
||||
<p>The API here is much simpler than the <a href="machi_flu1_client.html"><code>machi_flu1_client</code></a> or
|
||||
<a href="machi_proxy_flu1_client.html"><code>machi_proxy_flu1_client</code></a> APIs. This module's API is a
|
||||
proposed simple-but-complete form for clients who are not
|
||||
interested in being an active participant in a Machi cluster and to
|
||||
have the responsibility for Machi internals, i.e., client-side
|
||||
Chain Replication, client-side read repair, client-side tracking of
|
||||
internal Machi epoch & projection changes, etc.</p>
|
||||
|
||||
<p>This client is implemented as a long-lived Erlang process using
|
||||
<code>gen_server</code>-style OTP code practice. A naive client can expect
|
||||
that this process will manage all transient TCP session
|
||||
disconnections and Machi chain reconfigurations. This client's
|
||||
efforts are best-effort and can require some time to retry
|
||||
operations in certain failure cases, i.e., up to several seconds
|
||||
during a Machi projection & epoch change when a new server is
|
||||
added to the chain.</p>
|
||||
|
||||
<p>Doc TODO: Once this API stabilizes, add all relevant data type details
|
||||
to the EDoc here.</p>
|
||||
|
||||
|
||||
<h4><a name="Missing_API_features">Missing API features</a></h4>
|
||||
|
||||
<p>So far, there is one missing client API feature that ought to be
|
||||
added to Machi in the near future: more flexible checksum
|
||||
management.</p>
|
||||
|
||||
<p>Add a <code>source</code> annotation to all checksums to indicate where the
|
||||
checksum was calculated. For example,</p>
|
||||
|
||||
<ul>
|
||||
|
||||
<li> Calculated by client that performed the original chunk append,
|
||||
</li>
|
||||
|
||||
<li> Calculated by the 1st Machi server to receive an
|
||||
un-checksummed append request
|
||||
</li>
|
||||
|
||||
<li> Re-calculated by Machi to manage fewer checksums of blocks of
|
||||
data larger than the original client-specified chunks.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p>Client-side checksums would be the "strongest" type of
|
||||
checksum, meaning that any data corruption (of the original
|
||||
data and/or of the checksum itself) can be detected after the
|
||||
client-side calculation. There are too many horror stories on
|
||||
The Net about IP PDUs that are corrupted but unnoticed due to
|
||||
weak TCP checksums, buggy hardware, buggy OS drivers, etc.
|
||||
Checksum versioning is also desirable if/when the current checksum
|
||||
implementation changes from SHA-1 to something else.</p>
|
||||
|
||||
|
||||
<h4><a name="Implementation_notes">Implementation notes</a></h4>
|
||||
|
||||
<p>The major operation processing is implemented in a state machine-like
|
||||
manner. Before attempting an operation <code>X</code>, there's an initial
|
||||
operation <code>pre-X</code> that takes care of updating the epoch id,
|
||||
restarting client protocol proxies, and if there's any server
|
||||
instability (e.g. some server is wedged), then insert some sleep
|
||||
time. When the chain appears to have stabilized, then we try the <code>X</code>
|
||||
operation again.</p>
|
||||
|
||||
<p>Function name for the <code>pre-X</code> stuff is usually <code>X()</code>, and the
|
||||
function name for the <code>X</code> stuff is usually <code>X2()</code>. (I.e., the <code>X</code>
|
||||
stuff follows after <code>pre-X</code> and therefore has a <code>2</code> suffix on the
|
||||
function name.)</p>
|
||||
|
||||
In the case of read repair, there are two stages: find the value to
|
||||
perform the repair, then perform the repair writes. In the case of
|
||||
the repair writes, the <code>pre-X</code> function is named <code>read_repair3()</code>,
|
||||
and the <code>X</code> function is named <code>read_repair4()</code>.
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append_chunk-3">append_chunk/3</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk-4">append_chunk/4</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-4">append_chunk_extra/4</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-5">append_chunk_extra/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-2">checksum_list/2</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-3">checksum_list/3</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-1">list_files/1</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-2">list_files/2</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#quit-1">quit/1</a></td><td>Quit & close the connection to remote FLU and stop our
|
||||
proxy process.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-5">read_chunk/5</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-6">read_chunk/6</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td>Start a local, long-lived process that will be our steady
|
||||
& reliable communication proxy with the fickle & flaky
|
||||
remote Machi server.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-2">start_link/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-4">trim_chunk/4</a></td><td>Trim a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-5">trim_chunk/5</a></td><td>Trim a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-4">write_chunk/4</a></td><td>Write a chunk of data (that has already been
|
||||
allocated/sequenced by an earlier append_chunk_extra() call) to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-5">write_chunk/5</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-3">append_chunk/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec, Prefix, Chunk) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-4">append_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec, Prefix, Chunk, Timeout) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-4">append_chunk_extra/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(PidSpec, Prefix, Chunk, ChunkExtra) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-5">append_chunk_extra/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(PidSpec, Prefix, Chunk, ChunkExtra, Timeout0) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-2">checksum_list/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec, File) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-3">checksum_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec, File, Timeout0) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="list_files-1">list_files/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-2">list_files/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec, Timeout0) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="quit-1">quit/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>quit(PidSpec) -> any()</tt></p>
|
||||
</div><p>Quit & close the connection to remote FLU and stop our
|
||||
proxy process.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-5">read_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec, File, Offset, Size, Opts) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-6">read_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec, File, Offset, Size, Opts, Timeout0) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(P_srvr_list) -> any()</tt></p>
|
||||
</div><p>Start a local, long-lived process that will be our steady
|
||||
& reliable communication proxy with the fickle & flaky
|
||||
remote Machi server.</p>
|
||||
|
||||
<h3 class="function"><a name="start_link-2">start_link/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(P_srvr_list, Opts) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-4">trim_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec, File, Offset, Size) -> any()</tt></p>
|
||||
</div><p>Trim a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-5">trim_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec, File, Offset, Size, Timeout0) -> any()</tt></p>
|
||||
</div><p>Trim a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-4">write_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec, File, Offset, Chunk) -> any()</tt></p>
|
||||
</div><p>Write a chunk of data (that has already been
|
||||
allocated/sequenced by an earlier append_chunk_extra() call) to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-5">write_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec, File, Offset, Chunk, Timeout0) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
186
edoc/machi_csum_table.html
Normal file
186
edoc/machi_csum_table.html
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_csum_table</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_csum_table</h1>
|
||||
<ul class="index"><li><a href="#types">Data Types</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
|
||||
|
||||
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-byte_sequence">byte_sequence()</a></h3>
|
||||
<p><tt>byte_sequence() = {Offset::non_neg_integer(), Size::pos_integer() | infinity}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-table">table()</a></h3>
|
||||
<p><tt>table() = #machi_csum_table{}</tt></p>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#all-1">all/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#all_trimmed-2">all_trimmed/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#all_trimmed-3">all_trimmed/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#any_trimmed-3">any_trimmed/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#calc_unwritten_bytes-1">calc_unwritten_bytes/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#close-1">close/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#decode_csum_file_entry-1">decode_csum_file_entry/1</a></td><td>Decode a single <code>binary()</code> blob into an
|
||||
<code>{Offset,Size,TaggedCSum}</code> tuple.</td></tr>
|
||||
<tr><td valign="top"><a href="#delete-1">delete/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#encode_csum_file_entry-3">encode_csum_file_entry/3</a></td><td>Encode <code>Offset + Size + TaggedCSum</code> into an <code>iolist()</code> type for
|
||||
internal storage by the FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#encode_csum_file_entry_bin-3">encode_csum_file_entry_bin/3</a></td><td>Encode <code>Offset + Size + TaggedCSum</code> into an <code>binary()</code> type for
|
||||
internal storage by the FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#find-3">find/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#find_leftneighbor-2">find_leftneighbor/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#find_rightneighbor-2">find_rightneighbor/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#foldl_chunks-3">foldl_chunks/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#open-2">open/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#split_checksum_list_blob_decode-1">split_checksum_list_blob_decode/1</a></td><td>Split a <code>binary()</code> blob of <code>checksum_list</code> data into a list of
|
||||
<code>{Offset,Size,TaggedCSum}</code> tuples.</td></tr>
|
||||
<tr><td valign="top"><a href="#sync-1">sync/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim-3">trim/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim-5">trim/5</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write-4">write/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write-6">write/6</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="all-1">all/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>all(Machi_csum_table) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="all_trimmed-2">all_trimmed/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>all_trimmed(Machi_csum_table::<a href="#type-table">table()</a>, Pos::non_neg_integer()) -> boolean()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="all_trimmed-3">all_trimmed/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>all_trimmed(Machi_csum_table::<a href="#type-table">table()</a>, Left::non_neg_integer(), Right::non_neg_integer()) -> boolean()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="any_trimmed-3">any_trimmed/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>any_trimmed(CsumT::<a href="#type-table">table()</a>, Offset::pos_integer(), Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>) -> boolean()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="calc_unwritten_bytes-1">calc_unwritten_bytes/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>calc_unwritten_bytes(Machi_csum_table::<a href="#type-table">table()</a>) -> [<a href="#type-byte_sequence">byte_sequence()</a>]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="close-1">close/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>close(Machi_csum_table::<a href="#type-table">table()</a>) -> ok</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="decode_csum_file_entry-1">decode_csum_file_entry/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>decode_csum_file_entry(Else::binary()) -> error | {<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, <a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>}</tt><br></p>
|
||||
</div><p><p>Decode a single <code>binary()</code> blob into an
|
||||
<code>{Offset,Size,TaggedCSum}</code> tuple.</p>
|
||||
|
||||
<p>The internal encoding (which is currently exposed to the outside world
|
||||
via this function and related ones) is:</p>
|
||||
|
||||
<ul>
|
||||
<li> 1 byte: record length
|
||||
</li>
|
||||
<li> 8 bytes (unsigned big-endian): byte offset
|
||||
</li>
|
||||
<li> 4 bytes (unsigned big-endian): chunk size
|
||||
</li>
|
||||
<li> all remaining bytes: tagged checksum (1st byte = type tag)
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
See <code>machi.hrl</code> for the tagged checksum types, e.g.,
|
||||
<code>?CSUM_TAG_NONE</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="delete-1">delete/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete(Machi_csum_table::<a href="#type-table">table()</a>) -> ok</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="encode_csum_file_entry-3">encode_csum_file_entry/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>encode_csum_file_entry(Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, TaggedCSum::<a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>) -> iolist()</tt><br></p>
|
||||
</div><p>Encode <code>Offset + Size + TaggedCSum</code> into an <code>iolist()</code> type for
|
||||
internal storage by the FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="encode_csum_file_entry_bin-3">encode_csum_file_entry_bin/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>encode_csum_file_entry_bin(Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, TaggedCSum::<a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>) -> binary()</tt><br></p>
|
||||
</div><p>Encode <code>Offset + Size + TaggedCSum</code> into an <code>binary()</code> type for
|
||||
internal storage by the FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="find-3">find/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>find(Machi_csum_table::<a href="#type-table">table()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-file_size">machi_dt:file_size()</a>) -> [{<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-file_size">machi_dt:file_size()</a>, <a href="machi_dt.html#type-chunk_csum">machi_dt:chunk_csum()</a> | trimmed}]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="find_leftneighbor-2">find_leftneighbor/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>find_leftneighbor(CsumT::<a href="#type-table">table()</a>, Offset::non_neg_integer()) -> undefined | {non_neg_integer(), <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, trimmed | <a href="machi_dt.html#type-chunk_csum">machi_dt:chunk_csum()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="find_rightneighbor-2">find_rightneighbor/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>find_rightneighbor(CsumT::<a href="#type-table">table()</a>, Offset::non_neg_integer()) -> undefined | {non_neg_integer(), <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, trimmed | <a href="machi_dt.html#type-chunk_csum">machi_dt:chunk_csum()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="foldl_chunks-3">foldl_chunks/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>foldl_chunks(Fun::fun(({non_neg_integer(), non_neg_integer(), term()}, Acc0::term()) -> Acc::term()), Acc0::term(), Machi_csum_table::<a href="#type-table">table()</a>) -> Acc::term()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="open-2">open/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>open(CSumFilename::string(), Opts::<a href="proplists.html#type-proplist">proplists:proplist()</a>) -> {ok, <a href="#type-table">table()</a>} | {error, <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="split_checksum_list_blob_decode-1">split_checksum_list_blob_decode/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>split_checksum_list_blob_decode(Bin::binary()) -> {[{<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, <a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>}], TrailingJunk::binary()}</tt><br></p>
|
||||
</div><p>Split a <code>binary()</code> blob of <code>checksum_list</code> data into a list of
|
||||
<code>{Offset,Size,TaggedCSum}</code> tuples.</p>
|
||||
|
||||
<h3 class="function"><a name="sync-1">sync/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>sync(Machi_csum_table::<a href="#type-table">table()</a>) -> ok | {error, <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim-3">trim/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim(Machi_csum_table::<a href="#type-table">table()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-file_size">machi_dt:file_size()</a>) -> ok | {error, <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim-5">trim/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim(CsumT, Offset, Size, LeftUpdate, RightUpdate) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write-4">write/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(CsumT::<a href="#type-table">table()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-file_size">machi_dt:file_size()</a>, CSum::<a href="machi_dt.html#type-chunk_csum">machi_dt:chunk_csum()</a> | trimmed) -> ok | {error, trimmed | <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write-6">write/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(Machi_csum_table, Offset, Size, CSum, LeftUpdate, RightUpdate) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
110
edoc/machi_dt.html
Normal file
110
edoc/machi_dt.html
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_dt</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_dt</h1>
|
||||
<ul class="index"><li><a href="#types">Data Types</a></li></ul>
|
||||
|
||||
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk">chunk()</a></h3>
|
||||
<p><tt>chunk() = <a href="#type-chunk_bin">chunk_bin()</a> | {<a href="#type-chunk_csum">chunk_csum()</a>, <a href="#type-chunk_bin">chunk_bin()</a>}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_bin">chunk_bin()</a></h3>
|
||||
<p><tt>chunk_bin() = binary() | iolist()</tt></p>
|
||||
<p> client can use either</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_csum">chunk_csum()</a></h3>
|
||||
<p><tt>chunk_csum() = binary()</tt></p>
|
||||
<p> 1 byte tag, N-1 bytes checksum</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_pos">chunk_pos()</a></h3>
|
||||
<p><tt>chunk_pos() = {<a href="#type-file_offset">file_offset()</a>, <a href="#type-chunk_size">chunk_size()</a>, <a href="#type-file_name_s">file_name_s()</a>}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_s">chunk_s()</a></h3>
|
||||
<p><tt>chunk_s() = trimmed | binary()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_size">chunk_size()</a></h3>
|
||||
<p><tt>chunk_size() = non_neg_integer()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-chunk_summary">chunk_summary()</a></h3>
|
||||
<p><tt>chunk_summary() = {<a href="#type-file_offset">file_offset()</a>, <a href="#type-chunk_size">chunk_size()</a>, binary()}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-csum_tag">csum_tag()</a></h3>
|
||||
<p><tt>csum_tag() = none | client_sha | server_sha | server_regen_sha</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-epoch_csum">epoch_csum()</a></h3>
|
||||
<p><tt>epoch_csum() = binary()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-epoch_id">epoch_id()</a></h3>
|
||||
<p><tt>epoch_id() = {<a href="#type-epoch_num">epoch_num()</a>, <a href="#type-epoch_csum">epoch_csum()</a>}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-epoch_num">epoch_num()</a></h3>
|
||||
<p><tt>epoch_num() = -1 | non_neg_integer()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-error_general">error_general()</a></h3>
|
||||
<p><tt>error_general() = bad_arg | wedged | bad_checksum</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_info">file_info()</a></h3>
|
||||
<p><tt>file_info() = {<a href="#type-file_size">file_size()</a>, <a href="#type-file_name_s">file_name_s()</a>}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_name">file_name()</a></h3>
|
||||
<p><tt>file_name() = binary() | list()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_name_s">file_name_s()</a></h3>
|
||||
<p><tt>file_name_s() = binary()</tt></p>
|
||||
<p> server reply</p>
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_offset">file_offset()</a></h3>
|
||||
<p><tt>file_offset() = non_neg_integer()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_prefix">file_prefix()</a></h3>
|
||||
<p><tt>file_prefix() = binary() | list()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-file_size">file_size()</a></h3>
|
||||
<p><tt>file_size() = non_neg_integer()</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-inet_host">inet_host()</a></h3>
|
||||
<p><tt>inet_host() = <a href="inet.html#type-ip_address">inet:ip_address()</a> | <a href="inet.html#type-hostname">inet:hostname()</a></tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-inet_port">inet_port()</a></h3>
|
||||
<p><tt>inet_port() = <a href="inet.html#type-port_number">inet:port_number()</a></tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-projection">projection()</a></h3>
|
||||
<p><tt>projection() = #projection_v1{}</tt></p>
|
||||
|
||||
|
||||
<h3 class="typedecl"><a name="type-projection_type">projection_type()</a></h3>
|
||||
<p><tt>projection_type() = public | private</tt></p>
|
||||
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
118
edoc/machi_file_proxy.html
Normal file
118
edoc/machi_file_proxy.html
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_file_proxy</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_file_proxy</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>This is a proxy process which mediates access to Machi FLU
|
||||
controlled files.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>This is a proxy process which mediates access to Machi FLU
|
||||
controlled files. In particular, it manages the "write-once register"
|
||||
conceit at the heart of Machi's design.</p>
|
||||
|
||||
<p>Read, write and append requests for a single file will be managed
|
||||
through this proxy. Clients can also request syncs for specific
|
||||
types of filehandles.</p>
|
||||
|
||||
As operations are requested, the proxy keeps track of how many
|
||||
operations it has performed (and how many errors were generated.)
|
||||
After a sufficient number of inactivity, the server terminates
|
||||
itself.
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append-2">append/2</a></td><td>Append data.</td></tr>
|
||||
<tr><td valign="top"><a href="#append-4">append/4</a></td><td>Append data to file, supplying client metadata and (if desired) a
|
||||
reservation for additional space.</td></tr>
|
||||
<tr><td valign="top"><a href="#read-3">read/3</a></td><td>Read file at offset for length.</td></tr>
|
||||
<tr><td valign="top"><a href="#read-4">read/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-3">start_link/3</a></td><td>Start a new instance of the file proxy service.</td></tr>
|
||||
<tr><td valign="top"><a href="#stop-1">stop/1</a></td><td>Request to stop an instance of the file proxy service.</td></tr>
|
||||
<tr><td valign="top"><a href="#sync-1">sync/1</a></td><td>Force a sync of all filehandles.</td></tr>
|
||||
<tr><td valign="top"><a href="#sync-2">sync/2</a></td><td>Force a sync of a specific filehandle type.</td></tr>
|
||||
<tr><td valign="top"><a href="#trim-4">trim/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write-3">write/3</a></td><td>Write data at offset.</td></tr>
|
||||
<tr><td valign="top"><a href="#write-4">write/4</a></td><td>Write data at offset, including the client metadata.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append-2">append/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append(Pid::pid(), Data::binary()) -> {ok, File::string(), Offset::non_neg_integer()} | {error, term()}</tt><br></p>
|
||||
</div><p>Append data</p>
|
||||
|
||||
<h3 class="function"><a name="append-4">append/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append(Pid::pid(), ClientMeta::<a href="proplists.html#type-proplist">proplists:proplist()</a>, Extra::non_neg_integer(), Data::binary()) -> {ok, File::string(), Offset::non_neg_integer()} | {error, term()}</tt><br></p>
|
||||
</div><p>Append data to file, supplying client metadata and (if desired) a
|
||||
reservation for additional space. ClientMeta is a proplist and expects the
|
||||
same keys as write/4.</p>
|
||||
|
||||
<h3 class="function"><a name="read-3">read/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read(Pid::pid(), Offset::non_neg_integer(), Length::non_neg_integer()) -> {ok, [{Filename::string(), Offset::non_neg_integer(), Data::binary(), Checksum::binary()}]} | {error, Reason::term()}</tt><br></p>
|
||||
</div><p>Read file at offset for length. This returns a sequence of all
|
||||
written and trimmed (optional) bytes that overlaps with requested
|
||||
offset and length. Borders are not aligned.</p>
|
||||
|
||||
<h3 class="function"><a name="read-4">read/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read(Pid::pid(), Offset::non_neg_integer(), Length::non_neg_integer(), Opts::[{no_checksum | no_chunk | needs_trimmed, boolean()}]) -> {ok, [{Filename::string(), Offset::non_neg_integer(), Data::binary(), Checksum::binary()}]} | {error, Reason::term()}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-3">start_link/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName::atom(), Filename::string(), DataDir::string()) -> any()</tt><br></p>
|
||||
</div><p>Start a new instance of the file proxy service. Takes the filename
|
||||
and data directory as arguments. This function is typically called by the
|
||||
<code>machi_file_proxy_sup:start_proxy/2</code> function.</p>
|
||||
|
||||
<h3 class="function"><a name="stop-1">stop/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop(Pid::pid()) -> ok</tt><br></p>
|
||||
</div><p>Request to stop an instance of the file proxy service.</p>
|
||||
|
||||
<h3 class="function"><a name="sync-1">sync/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>sync(Pid::pid()) -> ok | {error, term()}</tt><br></p>
|
||||
</div><p>Force a sync of all filehandles</p>
|
||||
|
||||
<h3 class="function"><a name="sync-2">sync/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>sync(Pid::pid(), Type::all | data | csum) -> ok | {error, term()}</tt><br></p>
|
||||
</div><p>Force a sync of a specific filehandle type. Valid types are <code>all</code>, <code>csum</code> and <code>data</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="trim-4">trim/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim(Pid, Offset, Size, TriggerGC) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write-3">write/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(Pid::pid(), Offset::non_neg_integer(), Data::binary()) -> ok | {error, term()}</tt><br></p>
|
||||
</div><p>Write data at offset</p>
|
||||
|
||||
<h3 class="function"><a name="write-4">write/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(Pid::pid(), Offset::non_neg_integer(), ClientMeta::<a href="proplists.html#type-proplist">proplists:proplist()</a>, Data::binary()) -> ok | {error, term()}</tt><br></p>
|
||||
</div><p>Write data at offset, including the client metadata. ClientMeta is a proplist
|
||||
that expects the following keys and values:
|
||||
<ul>
|
||||
<li><code>client_csum_tag</code> - the type of checksum from the client as defined in the machi.hrl file</li>
|
||||
<li><code>client_csum</code> - the checksum value from the client</li>
|
||||
</ul></p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
51
edoc/machi_file_proxy_sup.html
Normal file
51
edoc/machi_file_proxy_sup.html
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_file_proxy_sup</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_file_proxy_sup</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>This is the main supervisor for the file proxies.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="supervisor.html"><tt>supervisor</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2>This is the main supervisor for the file proxies.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#child_spec-1">child_spec/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_proxy-3">start_proxy/3</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="child_spec-1">child_spec/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>child_spec(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_proxy-3">start_proxy/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_proxy(FluName, DataDir, Filename) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
120
edoc/machi_fitness.html
Normal file
120
edoc/machi_fitness.html
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_fitness</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_fitness</h1>
|
||||
<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#add_admin_down-3">add_admin_down/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#delete_admin_down-2">delete_admin_down/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#format_status-2">format_status/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_unfit_list-1">get_unfit_list/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#send_fitness_update_spam-3">send_fitness_update_spam/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#send_spam_to_everyone-1">send_spam_to_everyone/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trigger_early_adjustment-2">trigger_early_adjustment/2</a></td><td>For testing purposes, we don't want a test to wait for
|
||||
wall-clock time to elapse before the fitness server makes a
|
||||
down->up status decision.</td></tr>
|
||||
<tr><td valign="top"><a href="#update_local_down_list-3">update_local_down_list/3</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="add_admin_down-3">add_admin_down/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>add_admin_down(PidSpec, DownFLU, DownProps) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="delete_admin_down-2">delete_admin_down/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete_admin_down(PidSpec, DownFLU) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="format_status-2">format_status/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>format_status(Opt, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_unfit_list-1">get_unfit_list/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_unfit_list(PidSpec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(Args) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="send_fitness_update_spam-3">send_fitness_update_spam/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>send_fitness_update_spam(Pid, FromName, Dict) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="send_spam_to_everyone-1">send_spam_to_everyone/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>send_spam_to_everyone(Pid) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(Args) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trigger_early_adjustment-2">trigger_early_adjustment/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trigger_early_adjustment(Pid, FLU) -> any()</tt></p>
|
||||
</div><p>For testing purposes, we don't want a test to wait for
|
||||
wall-clock time to elapse before the fitness server makes a
|
||||
down->up status decision.</p>
|
||||
|
||||
<h3 class="function"><a name="update_local_down_list-3">update_local_down_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>update_local_down_list(PidSpec, Down, MembersDict) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
114
edoc/machi_flu1.html
Normal file
114
edoc/machi_flu1.html
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu1</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu1</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The Machi FLU file server + file location sequencer.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>The Machi FLU file server + file location sequencer.</p>
|
||||
|
||||
<p>This module implements only the Machi FLU file server and its
|
||||
implicit sequencer.
|
||||
Please see the EDoc "Overview" for details about the FLU as a
|
||||
primitive file server process vs. the larger Machi design of a FLU
|
||||
as a sequencer + file server + chain manager group of processes.</p>
|
||||
|
||||
<p>The FLU is named after the CORFU server "FLU" or "FLash Unit" server.</p>
|
||||
|
||||
<h4><a name="Protocol_origins">Protocol origins</a></h4>
|
||||
|
||||
<p>Today's FLU fully supports a protocol that is based on Protocol
|
||||
Buffers. Please see the <code>src/machi.proto</code> file for details.</p>
|
||||
|
||||
<h4><a name="TODO_items">TODO items</a></h4>
|
||||
|
||||
<p>TODO The per-file metadata tuple store is missing from this implementation.</p>
|
||||
|
||||
TODO Section 4.1 ("The FLU") of the Machi design doc suggests that
|
||||
the FLU keep track of the epoch number of the last file write (and
|
||||
perhaps last metadata write), as an optimization for inter-FLU data
|
||||
replication/chain repair.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#current_state-1">current_state/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#format_state-1">format_state/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#main2-4">main2/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_listener_regname-1">make_listener_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_projection_server_regname-1">make_projection_server_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#run_append_server-2">run_append_server/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#run_listen_server-1">run_listen_server/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop-1">stop/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#update_wedge_state-3">update_wedge_state/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_myself-2">wedge_myself/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="current_state-1">current_state/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>current_state(PidSpec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="format_state-1">format_state/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>format_state(State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="main2-4">main2/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>main2(FluName, TcpPort, DataDir, Props) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_listener_regname-1">make_listener_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_listener_regname(BaseName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_projection_server_regname-1">make_projection_server_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_projection_server_regname(BaseName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="run_append_server-2">run_append_server/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>run_append_server(FluPid, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="run_listen_server-1">run_listen_server/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>run_listen_server(State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(Rest) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop-1">stop/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop(Pid) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="update_wedge_state-3">update_wedge_state/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>update_wedge_state(PidSpec, Boolean, EpochId) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="wedge_myself-2">wedge_myself/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_myself(PidSpec, EpochId) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
340
edoc/machi_flu1_client.html
Normal file
340
edoc/machi_flu1_client.html
Normal file
|
|
@ -0,0 +1,340 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu1_client</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu1_client</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#types">Data Types</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Erlang API for the Machi FLU TCP protocol version 1.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Erlang API for the Machi FLU TCP protocol version 1.</p>
|
||||
|
||||
<p>This client API handles low-level PDU serialization/deserialization
|
||||
and low-level TCP session management, e.g. open, receive, write,
|
||||
close. The API for higher-level session management and Machi state
|
||||
management can be found in <a href="machi_proxy_flu1_client.html"><code>machi_proxy_flu1_client</code></a> and
|
||||
<a href="machi_cr_client.html"><code>machi_cr_client</code></a>.</p>
|
||||
|
||||
<p>For the moment, this module implements a Protocol Buffers-based
|
||||
protocol as the sole supported access method to the server,
|
||||
sequencer, and projection store. Conceptually, those three
|
||||
services are independent and ought to have their own protocols. As
|
||||
a practical matter, there is no need for wire protocol
|
||||
compatibility. Furthermore, from the perspective of failure
|
||||
detection, it is very convenient that all three FLU-related
|
||||
services are accessed using the same single TCP port.</p>
|
||||
|
||||
TODO This EDoc was written first, and the EDoc and also <code>-type</code> and
|
||||
<code>-spec</code> definitions for <a href="machi_proxy_flu1_client.html"><code>machi_proxy_flu1_client</code></a> and <a href="machi_cr_client.html"><code>machi_cr_client</code></a> must be improved.
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-port_wrap">port_wrap()</a></h3>
|
||||
<p><tt>port_wrap() = {w, atom(), term()}</tt></p>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append_chunk-4">append_chunk/4</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk-5">append_chunk/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-5">append_chunk_extra/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-6">append_chunk_extra/6</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-3">checksum_list/3</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-4">checksum_list/4</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#connect-1">connect/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#connected_p-1">connected_p/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#delete_migration-3">delete_migration/3</a></td><td>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</td></tr>
|
||||
<tr><td valign="top"><a href="#delete_migration-4">delete_migration/4</a></td><td>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</td></tr>
|
||||
<tr><td valign="top"><a href="#disconnect-1">disconnect/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#echo-2">echo/2</a></td><td>Echo -- test protocol round-trip.</td></tr>
|
||||
<tr><td valign="top"><a href="#echo-3">echo/3</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-2">get_all_projections/2</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-3">get_all_projections/3</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-2">get_latest_epochid/2</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-3">get_latest_epochid/3</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#kick_projection_reaction-2">kick_projection_reaction/2</a></td><td>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</td></tr>
|
||||
<tr><td valign="top"><a href="#kick_projection_reaction-3">kick_projection_reaction/3</a></td><td>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-2">list_all_projections/2</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-3">list_all_projections/3</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-2">list_files/2</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-3">list_files/3</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#quit-1">quit/1</a></td><td>Quit & close the connection to remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-6">read_chunk/6</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-7">read_chunk/7</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-2">read_latest_projection/2</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-3">read_latest_projection/3</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-3">read_projection/3</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-4">read_projection/4</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-5">trim_chunk/5</a></td><td>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#trunc_hack-3">trunc_hack/3</a></td><td>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</td></tr>
|
||||
<tr><td valign="top"><a href="#trunc_hack-4">trunc_hack/4</a></td><td>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-1">wedge_status/1</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-2">wedge_status/2</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-5">write_chunk/5</a></td><td>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-6">write_chunk/6</a></td><td>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-3">write_projection/3</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-4">write_projection/4</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-4">append_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Prefix::<a href="machi_dt.html#type-file_prefix">machi_dt:file_prefix()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>) -> {ok, <a href="machi_dt.html#type-chunk_pos">machi_dt:chunk_pos()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-5">append_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Prefix::<a href="machi_dt.html#type-file_prefix">machi_dt:file_prefix()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>) -> {ok, <a href="machi_dt.html#type-chunk_pos">machi_dt:chunk_pos()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-5">append_chunk_extra/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Prefix::<a href="machi_dt.html#type-file_prefix">machi_dt:file_prefix()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>, ChunkExtra::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>) -> {ok, <a href="machi_dt.html#type-chunk_pos">machi_dt:chunk_pos()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</p>
|
||||
|
||||
For example, if the <code>Chunk</code> size is 1 KByte and <code>Extra</code> is 4K Bytes, then
|
||||
the file offsets that follow <code>Chunk</code>'s position for the following 4K will
|
||||
be reserved by the file sequencer for later write(s) by the
|
||||
<code>write_chunk()</code> API.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-6">append_chunk_extra/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, Prefix::<a href="machi_dt.html#type-file_prefix">machi_dt:file_prefix()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>, ChunkExtra::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>) -> {ok, <a href="machi_dt.html#type-chunk_pos">machi_dt:chunk_pos()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</p>
|
||||
|
||||
For example, if the <code>Chunk</code> size is 1 KByte and <code>Extra</code> is 4K Bytes, then
|
||||
the file offsets that follow <code>Chunk</code>'s position for the following 4K will
|
||||
be reserved by the file sequencer for later write(s) by the
|
||||
<code>write_chunk()</code> API.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-3">checksum_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> {ok, binary()} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file | partial_read} | {error, term()}</tt><br></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-4">checksum_list/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> {ok, binary()} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file} | {error, term()}</tt><br></p>
|
||||
</div><p><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
Why return a simple <code>binary()</code> type rather than
|
||||
<code>[machi_dt:chunk_summary()]</code>? The two reasons are:
|
||||
<ol>
|
||||
<li> Server overhead: the CPU required to chop up the implementation-
|
||||
specific store into zillions of very small terms is very high.
|
||||
</li>
|
||||
<li> Protocol encoding and decoding overhead: the cost is non-zero,
|
||||
and the sum of cost of encoding and decoding a zillion small terms
|
||||
is substantial.
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<p>For both reasons, the server's protocol response is absurdly simple
|
||||
and very fast: send back a <code>binary()</code> blob to the client. Then it
|
||||
is the client's responsibility to spend the CPU time to parse the
|
||||
blob.</p>
|
||||
|
||||
Details of the encoding used inside the <code>binary()</code> blog can be found
|
||||
in the EDoc comments for <a href="machi_flu1.html#decode_csum_file_entry-1"><code>machi_flu1:decode_csum_file_entry/1</code></a>.</p>
|
||||
|
||||
<h3 class="function"><a name="connect-1">connect/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connect(P_srvr) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="connected_p-1">connected_p/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connected_p(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="delete_migration-3">delete_migration/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete_migration(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</p>
|
||||
|
||||
<h3 class="function"><a name="delete_migration-4">delete_migration/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete_migration(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</p>
|
||||
|
||||
<h3 class="function"><a name="disconnect-1">disconnect/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>disconnect(WS) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="echo-2">echo/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>echo(Sock::<a href="#type-port_wrap">port_wrap()</a>, String::string()) -> string() | {error, term()}</tt><br></p>
|
||||
</div><p>Echo -- test protocol round-trip.</p>
|
||||
|
||||
<h3 class="function"><a name="echo-3">echo/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>echo(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, String::string()) -> string() | {error, term()}</tt><br></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-2">get_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, [<a href="machi_dt.html#type-projection">machi_dt:projection()</a>]} | {error, term()}</tt><br></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-3">get_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, [<a href="machi_dt.html#type-projection">machi_dt:projection()</a>]} | {error, term()}</tt><br></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-2">get_latest_epochid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, <a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-3">get_latest_epochid/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, <a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="kick_projection_reaction-2">kick_projection_reaction/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>kick_projection_reaction(Sock::<a href="#type-port_wrap">port_wrap()</a>, Options::list()) -> ok</tt><br></p>
|
||||
</div><p>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</p>
|
||||
|
||||
<h3 class="function"><a name="kick_projection_reaction-3">kick_projection_reaction/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>kick_projection_reaction(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, Options::list()) -> ok</tt><br></p>
|
||||
</div><p>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-2">list_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, [non_neg_integer()]} | {error, term()}</tt><br></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-3">list_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, [non_neg_integer()]} | {error, term()}</tt><br></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-2">list_files/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>) -> {ok, [<a href="machi_dt.html#type-file_info">machi_dt:file_info()</a>]} | {error, term()}</tt><br></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-3">list_files/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>) -> {ok, [<a href="machi_dt.html#type-file_info">machi_dt:file_info()</a>]} | {error, term()}</tt><br></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="quit-1">quit/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>quit(Sock::<a href="#type-port_wrap">port_wrap()</a>) -> ok</tt><br></p>
|
||||
</div><p>Quit & close the connection to remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-6">read_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, Opts::<a href="proplists.html#type-proplist">proplists:proplist()</a>) -> {ok, <a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | not_written | partial_read} | {error, term()}</tt><br></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-7">read_chunk/7</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, Opts::<a href="proplists.html#type-proplist">proplists:proplist()</a>) -> {ok, <a href="machi_dt.html#type-chunk_s">machi_dt:chunk_s()</a>} | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | not_written | partial_read} | {error, term()}</tt><br></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-2">read_latest_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, <a href="machi_dt.html#type-projection">machi_dt:projection()</a>} | {error, not_written} | {error, term()}</tt><br></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-3">read_latest_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>) -> {ok, <a href="machi_dt.html#type-projection">machi_dt:projection()</a>} | {error, not_written} | {error, term()}</tt><br></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-3">read_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>, Epoch::<a href="machi_dt.html#type-epoch_num">machi_dt:epoch_num()</a>) -> {ok, <a href="machi_dt.html#type-projection">machi_dt:projection()</a>} | {error, not_written} | {error, term()}</tt><br></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-4">read_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>, Epoch::<a href="machi_dt.html#type-epoch_num">machi_dt:epoch_num()</a>) -> {ok, <a href="machi_dt.html#type-projection">machi_dt:projection()</a>} | {error, not_written} | {error, term()}</tt><br></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-5">trim_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File0::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="trunc_hack-3">trunc_hack/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trunc_hack(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</p>
|
||||
|
||||
<h3 class="function"><a name="trunc_hack-4">trunc_hack/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trunc_hack(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a> | no_such_file} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-1">wedge_status/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(Sock::<a href="#type-port_wrap">port_wrap()</a>) -> {ok, {boolean(), <a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>}} | {error, term()}</tt><br></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-2">wedge_status/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>) -> {ok, {boolean(), <a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>}} | {error, term()}</tt><br></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-5">write_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(Sock::<a href="#type-port_wrap">port_wrap()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-6">write_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, EpochID::<a href="machi_dt.html#type-epoch_id">machi_dt:epoch_id()</a>, File::<a href="machi_dt.html#type-file_name">machi_dt:file_name()</a>, Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Chunk::<a href="machi_dt.html#type-chunk">machi_dt:chunk()</a>) -> ok | {error, <a href="machi_dt.html#type-error_general">machi_dt:error_general()</a>} | {error, term()}</tt><br></p>
|
||||
</div><p>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-3">write_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(Sock::<a href="#type-port_wrap">port_wrap()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>, Proj::<a href="machi_dt.html#type-projection">machi_dt:projection()</a>) -> ok | {error, written} | {error, term()}</tt><br></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-4">write_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(Host::<a href="machi_dt.html#type-inet_host">machi_dt:inet_host()</a>, TcpPort::<a href="machi_dt.html#type-inet_port">machi_dt:inet_port()</a>, ProjType::<a href="machi_dt.html#type-projection_type">machi_dt:projection_type()</a>, Proj::<a href="machi_dt.html#type-projection">machi_dt:projection()</a>) -> ok | {error, written} | {error, term()}</tt><br></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
119
edoc/machi_flu_filename_mgr.html
Normal file
119
edoc/machi_flu_filename_mgr.html
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu_filename_mgr</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu_filename_mgr</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>This process is responsible for managing filenames assigned to
|
||||
prefixes.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>This process is responsible for managing filenames assigned to
|
||||
prefixes. It's started out of <code>machi_flu_psup</code>.</p>
|
||||
|
||||
<p>Supported operations include finding the "current" filename assigned to
|
||||
a prefix. Incrementing the sequence number and returning a new file name
|
||||
and listing all data files assigned to a given prefix.</p>
|
||||
|
||||
<p>All prefixes should have the form of <code>{prefix, P}</code>. Single filename
|
||||
return values have the form of <code>{file, F}</code>.</p>
|
||||
|
||||
<h2>Finding the current file associated with a sequence</h2><p>
|
||||
First it looks up the sequence number from the prefix name. If
|
||||
no sequence file is found, it uses 0 as the sequence number and searches
|
||||
for a matching file with the prefix and 0 as the sequence number.
|
||||
If no file is found, the it generates a new filename by incorporating
|
||||
the given prefix, a randomly generated (v4) UUID and 0 as the
|
||||
sequence number.</p>
|
||||
|
||||
If the sequence number is > 0, then the process scans the filesystem
|
||||
looking for a filename which matches the prefix and given sequence number and
|
||||
returns that.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#child_spec-2">child_spec/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#find_or_make_filename_from_prefix-3">find_or_make_filename_from_prefix/3</a></td><td>Find the latest available or make a filename from a prefix.</td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#increment_prefix_sequence-2">increment_prefix_sequence/2</a></td><td>Increment the sequence counter for a given prefix.</td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#list_files_by_prefix-2">list_files_by_prefix/2</a></td><td>Given a prefix in the form of <code>{prefix, P}</code> return
|
||||
all the data files associated with that prefix.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-2">start_link/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="child_spec-2">child_spec/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>child_spec(FluName, DataDir) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, State, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="find_or_make_filename_from_prefix-3">find_or_make_filename_from_prefix/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>find_or_make_filename_from_prefix(FluName::atom(), EpochId::<a href="#type-pv1_epoch_n">pv1_epoch_n()</a>, Prefix::{prefix, string()}) -> {file, Filename::string()} | {error, Reason::term()} | timeout</tt><br></p>
|
||||
</div><p>Find the latest available or make a filename from a prefix. A prefix
|
||||
should be in the form of a tagged tuple <code>{prefix, P}</code>. Returns a tagged
|
||||
tuple in the form of <code>{file, F}</code> or an <code>{error, Reason}</code></p>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Req, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Req, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="increment_prefix_sequence-2">increment_prefix_sequence/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>increment_prefix_sequence(FluName::atom(), Prefix::{prefix, string()}) -> ok | {error, Reason::term()} | timeout</tt><br></p>
|
||||
</div><p>Increment the sequence counter for a given prefix. Prefix should
|
||||
be in the form of <code>{prefix, P}</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="list_files_by_prefix-2">list_files_by_prefix/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files_by_prefix(FluName::atom(), Prefix::{prefix, string()}) -> [<a href="file.html#type-name">file:name()</a>] | timeout | {error, Reason::term()}</tt><br></p>
|
||||
</div><p>Given a prefix in the form of <code>{prefix, P}</code> return
|
||||
all the data files associated with that prefix. Returns
|
||||
a list.</p>
|
||||
|
||||
<h3 class="function"><a name="start_link-2">start_link/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName, DataDir) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, State) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
123
edoc/machi_flu_metadata_mgr.html
Normal file
123
edoc/machi_flu_metadata_mgr.html
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu_metadata_mgr</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu_metadata_mgr</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>This is a metadata service for the machi FLU which currently
|
||||
tracks the mappings between filenames and file proxies.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>This is a metadata service for the machi FLU which currently
|
||||
tracks the mappings between filenames and file proxies.</p>
|
||||
|
||||
<p>The service takes a given hash space and spreads it out over a
|
||||
pool of N processes which are responsible for 1/Nth the hash
|
||||
space. When a user requests an operation on a particular file
|
||||
the filename is hashed into the hash space and the request
|
||||
forwarded to a particular manager responsible for that slice
|
||||
of the hash space.</p>
|
||||
|
||||
The current hash implementation is <code>erlang:phash2/1</code> which has
|
||||
a range between 0..2^27-1 or 134,217,727.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#build_metadata_mgr_name-2">build_metadata_mgr_name/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#child_spec-4">child_spec/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#lookup_manager_pid-2">lookup_manager_pid/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#lookup_proxy_pid-2">lookup_proxy_pid/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-4">start_link/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_proxy_pid-2">start_proxy_pid/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop_proxy_pid-2">stop_proxy_pid/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim_file-2">trim_file/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="build_metadata_mgr_name-2">build_metadata_mgr_name/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>build_metadata_mgr_name(FluName, N) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="child_spec-4">child_spec/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>child_spec(FluName, C, DataDir, N) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, State, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Req, From, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Req, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="lookup_manager_pid-2">lookup_manager_pid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>lookup_manager_pid(FluName, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="lookup_proxy_pid-2">lookup_proxy_pid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>lookup_proxy_pid(FluName, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-4">start_link/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName, Name, DataDir, Num) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_proxy_pid-2">start_proxy_pid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_proxy_pid(FluName, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop_proxy_pid-2">stop_proxy_pid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop_proxy_pid(FluName, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim_file-2">trim_file/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_file(FluName, X2) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
49
edoc/machi_flu_metadata_mgr_sup.html
Normal file
49
edoc/machi_flu_metadata_mgr_sup.html
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu_metadata_mgr_sup</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu_metadata_mgr_sup</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>This is the supervisor for the collection of metadata
|
||||
managers.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="supervisor.html"><tt>supervisor</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2>This is the supervisor for the collection of metadata
|
||||
managers. It's started out of <code>machi_flu_psup</code>. It reads an
|
||||
application environment variable named <code>metadata_manager_count</code>
|
||||
with a default of 10 if it is not set.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#child_spec-3">child_spec/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-3">start_link/3</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="child_spec-3">child_spec/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>child_spec(FluName, DataDir, N) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-3">start_link/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName, DataDir, N) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
136
edoc/machi_flu_psup.html
Normal file
136
edoc/machi_flu_psup.html
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu_psup</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu_psup</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Supervisor for Machi FLU servers and their related support
|
||||
servers.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="supervisor.html"><tt>supervisor</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Supervisor for Machi FLU servers and their related support
|
||||
servers.</p>
|
||||
|
||||
<p>Our parent supervisor, <a href="machi_flu_sup.html"><code>machi_flu_sup</code></a>, is responsible for
|
||||
managing FLUs as a single entity. However, the actual
|
||||
implementation of a FLU includes three major Erlang processes (not
|
||||
including support/worker procs): the FLU itself, the FLU's
|
||||
projection store, and the FLU's local chain manager. This
|
||||
supervisor is responsible for managing those three major services
|
||||
as a single "package", to be started & stopped together.</p>
|
||||
|
||||
<p>The illustration below shows the OTP process supervision tree for
|
||||
the Machi application. Two FLUs are running, called <code>a</code> and <code>b</code>.
|
||||
The chain is configured for a third FLU, <code>c</code>, which is not running
|
||||
at this time.</p>
|
||||
|
||||
<p><img src="/machi//images/supervisor-2flus.png"></p>
|
||||
|
||||
<ul>
|
||||
<li> The FLU process itself is named <code>a</code>.
|
||||
</li>
|
||||
<li> The projection store process is named <code>a_pstore</code>.
|
||||
</li>
|
||||
<li> The chain manager process is named <code>a_chmgr</code>. The three
|
||||
linked subprocesses are long-lived <a href="machi_proxy_flu1_client.html"><code>machi_proxy_flu1_client</code></a> processes for communicating to all
|
||||
chain participants' projection stores (including the local
|
||||
store <code>a_pstore</code>).
|
||||
</li>
|
||||
<li> A fourth major process, <code>a_listener</code>, which is responsible for
|
||||
listening on a TCP socket and creating new connections.
|
||||
Currently, each listener has two processes handling incoming
|
||||
requests, one from each chain manager proxy.
|
||||
</li>
|
||||
<li> Note that the sub-supervisor parent of <code>a</code> and <code>a_listener</code> does
|
||||
not have a registered name.
|
||||
</li>
|
||||
</ul>
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_fitness_regname-1">make_fitness_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_flu_regname-1">make_flu_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_mgr_supname-1">make_mgr_supname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_p_regname-1">make_p_regname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_package_spec-1">make_package_spec/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_package_spec-4">make_package_spec/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_proj_supname-1">make_proj_supname/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_flu_package-1">start_flu_package/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_flu_package-4">start_flu_package/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-4">start_link/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop_flu_package-1">stop_flu_package/1</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_fitness_regname-1">make_fitness_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_fitness_regname(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_flu_regname-1">make_flu_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_flu_regname(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_mgr_supname-1">make_mgr_supname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_mgr_supname(MgrName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_p_regname-1">make_p_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_p_regname(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_package_spec-1">make_package_spec/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_package_spec(P_srvr) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_package_spec-4">make_package_spec/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_package_spec(FluName, TcpPort, DataDir, Props) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_proj_supname-1">make_proj_supname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_proj_supname(ProjName) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_flu_package-1">start_flu_package/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_flu_package(P_srvr) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_flu_package-4">start_flu_package/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_flu_package(FluName, TcpPort, DataDir, Props) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-4">start_link/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(FluName, TcpPort, DataDir, Props) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop_flu_package-1">stop_flu_package/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop_flu_package(FluName) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
53
edoc/machi_flu_sup.html
Normal file
53
edoc/machi_flu_sup.html
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_flu_sup</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_flu_sup</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Supervisor for Machi FLU servers and their related support
|
||||
servers.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="supervisor.html"><tt>supervisor</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Supervisor for Machi FLU servers and their related support
|
||||
servers.</p>
|
||||
|
||||
<p>Responsibility for managing FLU and chain lifecycle after the initial
|
||||
application startup is delegated to <a href="machi_lifecycle_mgr.html"><code>machi_lifecycle_mgr</code></a>.</p>
|
||||
|
||||
See <a href="machi_flu_psup.html"><code>machi_flu_psup</code></a> for an illustration of the entire Machi
|
||||
application process structure.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#load_rc_d_files_from_dir-1">load_rc_d_files_from_dir/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="load_rc_d_files_from_dir-1">load_rc_d_files_from_dir/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>load_rc_d_files_from_dir(Dir) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-0">start_link/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link() -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
229
edoc/machi_lifecycle_mgr.html
Normal file
229
edoc/machi_lifecycle_mgr.html
Normal file
|
|
@ -0,0 +1,229 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_lifecycle_mgr</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_lifecycle_mgr</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Lifecycle manager for Machi FLUs and chains.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Lifecycle manager for Machi FLUs and chains.</p>
|
||||
|
||||
<p>Over the lifetime of a Machi cluster, both the number and types of
|
||||
FLUs and chains may change. The lifecycle manager is responsible
|
||||
for implementing the lifecycle changes as expressed by "policy".
|
||||
In our case, "policy" is created by an external administrative
|
||||
entity that creates and deletes configuration files that define
|
||||
FLUs and chains relative to this local machine.</p>
|
||||
|
||||
<p>The "master configuration" for deciding which FLUs should be
|
||||
running on this machine was inspired by BSD UNIX's <code>init(8)</code> and the
|
||||
"rc.d" scheme. FLU definitions are found in a single directory,
|
||||
with one file per FLU. Chains are defined similarly, with one
|
||||
definition file per chain.</p>
|
||||
|
||||
<p>If a definition file for a FLU (or chain) exists, then that
|
||||
FLU/chain ought to be configured into being and running. If a
|
||||
definition file for a FLU/chain is removed, then that FLU/chain
|
||||
should be stopped gracefully. However, deleting of a file destroys
|
||||
information that is stored inside of that file. Therefore, we will
|
||||
<b>not allow arbitrary unlinking of lifecycle config files</b>. If
|
||||
the administrator deletes these config files using <code>unlink(8)</code>
|
||||
directly, then "the warranty has been broken".</p>
|
||||
|
||||
<p>We will rely on using an administrative command to inform the
|
||||
running system to stop and/or delete lifecycle resources. If the
|
||||
Machi application is not running, sorry, please start Machi first.</p>
|
||||
|
||||
<h3><a name="Wheel_reinvention">Wheel reinvention</a></h3>
|
||||
|
||||
<p>There's a whole mess of configuration management research &
|
||||
libraries out there. I hope to ignore them all by doing something
|
||||
quick & dirty & good enough here. If I fail, then I'll go
|
||||
pay attention to That Other Stuff.</p>
|
||||
|
||||
<h3><a name="A_note_about_policy">A note about policy</a></h3>
|
||||
|
||||
<p>It is outside of the scope of this local lifecycle manager to make
|
||||
decisions about policy or to distribute policy info/files/whatever
|
||||
to other machines. This is our machine. There are many like it,
|
||||
but this one is ours.</p>
|
||||
|
||||
<h3><a name="Machi_Application_Variables">Machi Application Variables</a></h3>
|
||||
|
||||
<p>All OTP application environment variables below are defined in the
|
||||
<code>machi</code> application.</p>
|
||||
|
||||
<ul>
|
||||
<li> <tt>flu_config_dir</tt>: Stores the <code>rc.d-</code>like config files for
|
||||
FLU runtime policy.
|
||||
</li>
|
||||
<li> <tt>flu_data_dir</tt>: Stores the file data and metadata for
|
||||
all FLUs.
|
||||
</li>
|
||||
<li> <tt>chain_config_dir</tt>: Stores the <code>rc.d</code>-like config files for
|
||||
chain runtime policy.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<h3><a name="The_FLU_Lifecycle">The FLU Lifecycle</a></h3>
|
||||
|
||||
<p>FLUs on the local machine may be started and stopped, as defined by
|
||||
administrative policy. In order to do any useful work, however, a
|
||||
running FLU must also be configured to be a member of a replication
|
||||
chain. Thus, as a practical matter, both a FLU and the chain that
|
||||
the FLU participates in must both be managed by this manager.</p>
|
||||
|
||||
<p>When a new <code>rc.d</code>-style config file is written to the FLU
|
||||
definition directory, a Machi server process will discover the file
|
||||
within a certain period of time, e.g. 15 seconds. The FLU will be
|
||||
started with the file's specified parameters. A FLU should be
|
||||
defined and started before configuring its chain membership.</p>
|
||||
|
||||
<p>Usually a FLU is removed implicitly by removing that FLU from the a
|
||||
newer definition file for the chain, or by deleting the entire
|
||||
chain definition. If a FLU has been started but never been a chain
|
||||
member, then the FLU can be stopped & removed explicitly.</p>
|
||||
|
||||
<p>When a FLU has been removed by policy, the FLU's data files are set
|
||||
aside into a temporary area. An additional policy command may be
|
||||
used to permanently delete such FLUs' data files, i.e. to reclaim
|
||||
disk space.</p>
|
||||
|
||||
<p>Resources for the FLU are defined in <code>machi_projection.hrl</code>
|
||||
in the <code>p_srvr{}</code> record. The major elements of this record are:</p>
|
||||
|
||||
<ul>
|
||||
|
||||
<li> <tt>name :: atom()</tt>: The name of the FLU. This name
|
||||
should be unique over the lifetime of the administrative
|
||||
domain and thus managed by outside policy. This name must be
|
||||
the same as the name of the <code>rc.d</code>-style config file that
|
||||
defines the FLU.
|
||||
</li>
|
||||
<li> <tt>address :: string()</tt>: The DNS hostname or IP address
|
||||
used by other servers to communicate with this FLU.
|
||||
</li>
|
||||
<li> <tt>port :: non_neg_integer() </tt>: The TCP port number that
|
||||
the FLU listens to for incoming Protocol Buffers-serialized
|
||||
communication.
|
||||
</li>
|
||||
<li> <tt>props :: property_list()</tt>: A general-purpose property
|
||||
list. Its use is currently fluid & not well-defined yet.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<h3><a name="The_Chain_Lifecycle">The Chain Lifecycle</a></h3>
|
||||
|
||||
<p>If a FLU on the local machine is expected to participate in a
|
||||
replication chain, then an <code>rc.d</code>-style chain definition file must
|
||||
also be present on each machine that runs a FLU in the chain.</p>
|
||||
|
||||
<p>Machi's chains are self-managing, via Humming Consensus; see the
|
||||
<a href="https://github.com/basho/machi/tree/master/doc/" target="_top"><tt>https://github.com/basho/machi/tree/master/doc/</tt></a> directory for
|
||||
much more detail about Humming Consensus. After FLUs have received
|
||||
their initial chain configuration for Humming Consensus, the FLUs
|
||||
will manage each other (and the chain) themselves.</p>
|
||||
|
||||
<p>However, Humming Consensus does not handle three chain management
|
||||
problems: 1. specifying the very first chain configuration,
|
||||
2. altering the membership of the chain (adding/removing FLUs from
|
||||
the chain), or 3. stopping the chain permanently.</p>
|
||||
|
||||
<p>FLUs in a new chain should have definition files created on each
|
||||
FLU's respective machine prior to defining their chain. Similarly,
|
||||
on each machine that hosts a chain member, a chain definition file
|
||||
created. External policy is responsible for creating each of these
|
||||
files.</p>
|
||||
|
||||
<p>Resources for the chain are defined in <code>machi_projection.hrl</code>
|
||||
in the <code>chain_def_v1{}</code> record. The major elements of this record are:</p>
|
||||
|
||||
<ul>
|
||||
|
||||
<li> <tt>name :: atom()</tt>: The name of the chain. This name
|
||||
should be unique over the lifetime of the administrative
|
||||
domain and thus managed by outside policy. This name must be
|
||||
the same as the name of the <code>rc.d</code>-style config file that
|
||||
defines the chain.
|
||||
</li>
|
||||
<li> <tt>mode :: 'ap_mode' | 'cp_mode'</tt>: This is the consistency
|
||||
to be used for managing the chain's replicated data: eventual
|
||||
consistency and strong consistency, respectively.
|
||||
</li>
|
||||
<li> <tt>full :: [#p_srvr{}] </tt>: A list of <code>#p_srvr{}</code> records
|
||||
to define the full-service members of the chain.
|
||||
</li>
|
||||
<li> <tt>witnesses :: [#p_srvr{}] </tt>: A list of <code>#p_srvr{}</code> records
|
||||
to define the witness-only members of the chain. Witness servers
|
||||
may only be used with strong consistency mode.
|
||||
</li>
|
||||
<li> <tt>props :: property_list()</tt>: A general-purpose property
|
||||
list. Its use is currently fluid & not well-defined yet.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<h3><a name="Conflicts_with_TCP_ports,_FLU_&_chain_names,_etc">Conflicts with TCP ports, FLU & chain names, etc</a></h3>
|
||||
|
||||
This manager is not responsible for managing conflicts in resource
|
||||
namespaces, e.g., TCP port numbers, FLU names, chain names, etc.
|
||||
Managing these namespaces is external policy's responsibility.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, State, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, State) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-0">start_link/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link() -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, State) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
69
edoc/machi_merkle_tree.html
Normal file
69
edoc/machi_merkle_tree.html
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_merkle_tree</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_merkle_tree</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Creates a Merkle tree per file based on the checksum data for
|
||||
a given data file.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Creates a Merkle tree per file based on the checksum data for
|
||||
a given data file.</p>
|
||||
|
||||
<p>The <code>naive</code> implementation representation is:</p>
|
||||
|
||||
<p><code><<Length:64, Offset:32, 0>></code> for unwritten bytes
|
||||
<code><<Length:64, Offset:32, 1>></code> for trimmed bytes
|
||||
<code><<Length:64, Offset:32, Csum/binary>></code> for written bytes</p>
|
||||
|
||||
The tree feeds these leaf nodes into hashes representing chunks of a minimum
|
||||
size of at least 1024 KB (1 MB), but if the file size is larger, we will try
|
||||
to get about 100 chunks for the first rollup "Level 1." We aim for around 10
|
||||
hashes at level 2, and then 2 hashes level 3 and finally the root.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#diff-2">diff/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#filename-1">filename/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#open-2">open/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#open-3">open/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#tree-1">tree/1</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="diff-2">diff/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>diff(Mt, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="filename-1">filename/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>filename(Mt) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="open-2">open/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>open(Filename, DataDir) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="open-3">open/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>open(Filename, DataDir, Type) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="tree-1">tree/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>tree(Mt) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
204
edoc/machi_pb_high_client.html
Normal file
204
edoc/machi_pb_high_client.html
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_pb_high_client</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_pb_high_client</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#types">Data Types</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Machi PB (Protocol Buffers) high-level client (prototype, API TBD).
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Machi PB (Protocol Buffers) high-level client (prototype, API TBD)</p>
|
||||
|
||||
At the moment, this is brittle-prototype-quality code: the API is not yet
|
||||
fully specified, there is very little error handling with respect
|
||||
to a single socket connection, and there is no code to deal with
|
||||
multiple connections/load balancing/error handling to several/all
|
||||
Machi cluster servers.
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-machi_client_error_reason">machi_client_error_reason()</a></h3>
|
||||
<p><tt>machi_client_error_reason() = bad_arg | wedged | bad_checksum | partition | not_written | written | trimmed | no_such_file | partial_read | bad_epoch | <a href="inet.html#type-posix">inet:posix()</a></tt></p>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append_chunk-6">append_chunk/6</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk-7">append_chunk/7</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#auth-3">auth/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#auth-4">auth/4</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-2">checksum_list/2</a></td><td>Returns a binary that has checksums and chunks encoded inside
|
||||
(This is because encoding-decoding them are inefficient).</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-3">checksum_list/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#connected_p-1">connected_p/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#convert_general_status_code-1">convert_general_status_code/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#echo-2">echo/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#echo-3">echo/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-1">list_files/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-2">list_files/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#quit-1">quit/1</a></td><td>Stops a client process.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-5">read_chunk/5</a></td><td>Tries to read a chunk of a specified file.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-6">read_chunk/6</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td>Creates a client process.</td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-4">trim_chunk/4</a></td><td>Trims arbitrary binary range of any file.</td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-5">trim_chunk/5</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-5">write_chunk/5</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-6">write_chunk/6</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-6">append_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec::pid(), PlacementKey::binary(), Prefix::binary(), Chunk::binary(), CSum::binary(), ChunkExtra::non_neg_integer()) -> {ok, Filename::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-7">append_chunk/7</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec::pid(), PlacementKey::binary(), Prefix::binary(), Chunk::binary(), CSum::binary(), ChunkExtra::non_neg_integer(), Timeout::non_neg_integer()) -> {ok, Filename::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="auth-3">auth/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>auth(PidSpec::pid(), User::string(), Pass::string()) -> ok | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="auth-4">auth/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>auth(PidSpec::pid(), User::string(), Pass::string(), Timeout::non_neg_integer()) -> ok | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-2">checksum_list/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec::pid(), File::string()) -> {ok, binary()} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div><p>Returns a binary that has checksums and chunks encoded inside
|
||||
(This is because encoding-decoding them are inefficient). TODO:
|
||||
return a structured list of them.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-3">checksum_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec, File, Timeout) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="connected_p-1">connected_p/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connected_p(PidSpec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="convert_general_status_code-1">convert_general_status_code/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>convert_general_status_code(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="echo-2">echo/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>echo(PidSpec::pid(), String::string()) -> {ok, string()} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="echo-3">echo/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>echo(PidSpec::pid(), String::string(), Timeout::non_neg_integer()) -> {ok, string()} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="list_files-1">list_files/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="list_files-2">list_files/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec, Timeout) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="quit-1">quit/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>quit(PidSpec::pid()) -> ok</tt><br></p>
|
||||
</div><p>Stops a client process.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-5">read_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec::pid(), File::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, Options::[{flag_no_checksum | flag_no_chunk | needs_trimmed, boolean()}]) -> {ok, {Chunks::[{File::string(), <a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, binary()}], Trimmed::[{File::string(), <a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>}]}} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div><p>Tries to read a chunk of a specified file. It returns <code>{ok,
|
||||
{Chunks, TrimmedChunks}}</code> for live file while it returns <code>{error,
|
||||
trimmed}</code> if all bytes of the file was trimmed.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-6">read_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec::pid(), File::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, Options::[{flag_no_checksum | flag_no_chunk | needs_trimmed, boolean()}], Timeout::non_neg_integer()) -> {ok, {Chunks::[{File::string(), <a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>, binary()}], Trimmed::[{File::string(), <a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, <a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>}]}} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(P_srvr_list::<a href="#type-p_srvr_dict">p_srvr_dict()</a>) -> {ok, pid()} | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div><p>Creates a client process</p>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-4">trim_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec::pid(), File::string(), Offset::non_neg_integer(), Size::<a href="machi_dt.html#type-chunk_size">machi_dt:chunk_size()</a>) -> ok | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div><p>Trims arbitrary binary range of any file. If a specified range
|
||||
has any byte trimmed, it fails and returns <code>{error, trimmed}</code>.
|
||||
Otherwise it trims all bytes in that range. If there are
|
||||
overlapping chunks with client-specified checksum, they will cut
|
||||
off and checksum are re-calculated in server side. TODO: Add
|
||||
option specifying whether to trigger GC.</p>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-5">trim_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec, File, Offset, Size, Timeout) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-5">write_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec::pid(), File::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Chunk::binary(), CSum::binary()) -> ok | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-6">write_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec::pid(), File::string(), Offset::<a href="machi_dt.html#type-file_offset">machi_dt:file_offset()</a>, Chunk::binary(), CSum::binary(), Timeout::non_neg_integer()) -> ok | {error, <a href="#type-machi_client_error_reason">machi_client_error_reason()</a>}</tt><br></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
31
edoc/machi_pb_server.html
Normal file
31
edoc/machi_pb_server.html
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_pb_server</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_pb_server</h1>
|
||||
<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#run_loop-2">run_loop/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="run_loop-2">run_loop/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>run_loop(Sock, P_srvr_list) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Jun 25 2015, 16:39:10.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
55
edoc/machi_pb_translate.html
Normal file
55
edoc/machi_pb_translate.html
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_pb_translate</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_pb_translate</h1>
|
||||
<ul class="index"><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#conv_to_csum_tag-1">conv_to_csum_tag/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#from_pb_request-1">from_pb_request/1</a></td><td>Adapt impedence mismatches between Erlang and Protocol Buffers.</td></tr>
|
||||
<tr><td valign="top"><a href="#from_pb_response-1">from_pb_response/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#to_pb_request-2">to_pb_request/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#to_pb_response-3">to_pb_response/3</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="conv_to_csum_tag-1">conv_to_csum_tag/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>conv_to_csum_tag(Type) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="from_pb_request-1">from_pb_request/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>from_pb_request(Mpb_ll_request) -> any()</tt></p>
|
||||
</div><p>Adapt impedence mismatches between Erlang and Protocol Buffers.</p>
|
||||
|
||||
<h3 class="function"><a name="from_pb_response-1">from_pb_response/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>from_pb_response(Mpb_ll_response) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="to_pb_request-2">to_pb_request/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>to_pb_request(ReqID, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="to_pb_response-3">to_pb_response/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>to_pb_response(ReqID, X2, X) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
77
edoc/machi_pb_wrap.html
Normal file
77
edoc/machi_pb_wrap.html
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_pb_wrap</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_pb_wrap</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Wrappers for Protocol Buffers encoding, including hacks to fix
|
||||
impedance mismatches between Erlang terms and PB encodings.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>Wrappers for Protocol Buffers encoding, including hacks to fix
|
||||
impedance mismatches between Erlang terms and PB encodings.
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#dec_p_srvr-1">dec_p_srvr/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#dec_projection_v1-1">dec_projection_v1/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#enc_p_srvr-1">enc_p_srvr/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#enc_projection_v1-1">enc_projection_v1/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_projection_req-2">make_projection_req/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_projection_resp-3">make_projection_resp/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#unmake_projection_req-1">unmake_projection_req/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#unmake_projection_resp-1">unmake_projection_resp/1</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="dec_p_srvr-1">dec_p_srvr/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>dec_p_srvr(Bin) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="dec_projection_v1-1">dec_projection_v1/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>dec_projection_v1(Bin) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="enc_p_srvr-1">enc_p_srvr/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>enc_p_srvr(P) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="enc_projection_v1-1">enc_projection_v1/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>enc_projection_v1(P) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_projection_req-2">make_projection_req/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_projection_req(ID, X2) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_projection_resp-3">make_projection_resp/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_projection_resp(ID, X2, Status) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="unmake_projection_req-1">unmake_projection_req/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>unmake_projection_req(Mpb_ll_request) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="unmake_projection_resp-1">unmake_projection_resp/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>unmake_projection_resp(Mpb_ll_response) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Jun 25 2015, 16:39:10.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
61
edoc/machi_plist.html
Normal file
61
edoc/machi_plist.html
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_plist</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_plist</h1>
|
||||
<ul class="index"><li><a href="#types">Data Types</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>
|
||||
|
||||
|
||||
<h2><a name="types">Data Types</a></h2>
|
||||
|
||||
<h3 class="typedecl"><a name="type-plist">plist()</a></h3>
|
||||
<p><tt>plist() = #machi_plist{}</tt></p>
|
||||
|
||||
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#add-2">add/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#all-1">all/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#close-1">close/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#find-2">find/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#open-2">open/2</a></td><td>persistent list of binaries.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="add-2">add/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>add(Plist::<a href="#type-plist">plist()</a>, Name::string()) -> {ok, <a href="#type-plist">plist()</a>} | {error, <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="all-1">all/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>all(Machi_plist::<a href="#type-plist">plist()</a>) -> [<a href="file.html#type-filename">file:filename()</a>]</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="close-1">close/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>close(Machi_plist::<a href="#type-plist">plist()</a>) -> ok</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="find-2">find/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>find(Machi_plist::<a href="#type-plist">plist()</a>, Name::string()) -> boolean()</tt><br></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="open-2">open/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>open(Filename::<a href="file.html#type-filename_all">file:filename_all()</a>, Opt::<a href="proplists.html#type-proplist">proplists:proplist()</a>) -> {ok, <a href="#type-plist">plist()</a>} | {error, <a href="file.html#type-posix">file:posix()</a>}</tt><br></p>
|
||||
</div><p>persistent list of binaries</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
106
edoc/machi_projection.html
Normal file
106
edoc/machi_projection.html
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_projection</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_projection</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>API for manipulating Machi projection data structures (i.e., records).
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>API for manipulating Machi projection data structures (i.e., records).
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#compare-2">compare/2</a></td><td>Compare two projection records for equality (assuming that the
|
||||
checksum element has been correctly calculated).</td></tr>
|
||||
<tr><td valign="top"><a href="#get_epoch_id-1">get_epoch_id/1</a></td><td>Return the epoch_id of the projection.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_epoch_id-1">make_epoch_id/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#make_members_dict-1">make_members_dict/1</a></td><td>Make a <code>p_srvr_dict()</code> out of a list of <code>p_srvr()</code> or out of a
|
||||
<code>p_srvr_dict()</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_summary-1">make_summary/1</a></td><td>Create a proplist-style summary of a projection record.</td></tr>
|
||||
<tr><td valign="top"><a href="#new-6">new/6</a></td><td>Create a new projection record.</td></tr>
|
||||
<tr><td valign="top"><a href="#new-7">new/7</a></td><td>Create a new projection record.</td></tr>
|
||||
<tr><td valign="top"><a href="#new-8">new/8</a></td><td>Create a new projection record.</td></tr>
|
||||
<tr><td valign="top"><a href="#update_checksum-1">update_checksum/1</a></td><td>Update the checksum element of a projection record.</td></tr>
|
||||
<tr><td valign="top"><a href="#update_dbg2-2">update_dbg2/2</a></td><td>Update the <code>dbg2</code> element of a projection record.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="compare-2">compare/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>compare(Projection_v1::#projection_v1{}, Projection_v1::#projection_v1{}) -> integer()</tt><br></p>
|
||||
</div><p><p>Compare two projection records for equality (assuming that the
|
||||
checksum element has been correctly calculated).</p>
|
||||
|
||||
The name "compare" is probably too close to "rank"? This
|
||||
comparison has nothing to do with projection ranking.</p>
|
||||
|
||||
<h3 class="function"><a name="get_epoch_id-1">get_epoch_id/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_epoch_id(Projection_v1) -> any()</tt></p>
|
||||
</div><p>Return the epoch_id of the projection.</p>
|
||||
|
||||
<h3 class="function"><a name="make_epoch_id-1">make_epoch_id/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_epoch_id(Projection_v1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="make_members_dict-1">make_members_dict/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_members_dict(Ps::[<a href="#type-p_srvr">p_srvr()</a>] | <a href="#type-p_srvr_dict">p_srvr_dict()</a>) -> <a href="#type-p_srvr_dict">p_srvr_dict()</a></tt><br></p>
|
||||
</div><p><p>Make a <code>p_srvr_dict()</code> out of a list of <code>p_srvr()</code> or out of a
|
||||
<code>p_srvr_dict()</code>.</p>
|
||||
|
||||
<p>If <code>Ps</code> is a <code>p_srvr_dict()</code>, then this function is usually a
|
||||
no-op. However, if someone has tampered with the list and screwed
|
||||
up its order, then we should fix it so <code>orddict</code> can work
|
||||
correctly.</p>
|
||||
|
||||
If <code>Ps</code> is simply <code>list(p_srvr())</code>, in which case we'll convert it
|
||||
to a <code>p_srvr_dict()</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="make_summary-1">make_summary/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_summary(Projection_v1) -> any()</tt></p>
|
||||
</div><p>Create a proplist-style summary of a projection record.</p>
|
||||
|
||||
<h3 class="function"><a name="new-6">new/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>new(MyName, MemberDict, Down_list, UPI_list, Repairing_list, Ps) -> any()</tt></p>
|
||||
</div><p>Create a new projection record.</p>
|
||||
|
||||
<h3 class="function"><a name="new-7">new/7</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>new(EpochNum, MyName, MemberDict, Down_list, UPI_list, Repairing_list, Dbg) -> any()</tt></p>
|
||||
</div><p>Create a new projection record.</p>
|
||||
|
||||
<h3 class="function"><a name="new-8">new/8</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>new(EpochNum, MyName, MembersDict0, Down_list, UPI_list, Repairing_list, Dbg, Dbg2) -> any()</tt></p>
|
||||
</div><p><p>Create a new projection record.</p>
|
||||
|
||||
The <code>MemberDict0</code> argument may be a true <code>p_srvr_dict()</code> (i.e, it
|
||||
is a well-formed <code>orddict</code> with the correct 2-tuple key-value form)
|
||||
or it may be simply <code>list(p_srvr())</code>, in which case we'll convert it
|
||||
to a <code>p_srvr_dict()</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="update_checksum-1">update_checksum/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>update_checksum(P) -> any()</tt></p>
|
||||
</div><p>Update the checksum element of a projection record.</p>
|
||||
|
||||
<h3 class="function"><a name="update_dbg2-2">update_dbg2/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>update_dbg2(P, Dbg2) -> any()</tt></p>
|
||||
</div><p>Update the <code>dbg2</code> element of a projection record.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
181
edoc/machi_projection_store.html
Normal file
181
edoc/machi_projection_store.html
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_projection_store</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_projection_store</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The Machi write-once projection store service.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>The Machi write-once projection store service.</p>
|
||||
|
||||
<p>This API is gen_server-style message passing, intended for use
|
||||
within a single Erlang node to glue together the projection store
|
||||
server with the node-local process that implements Machi's FLU
|
||||
client access protocol (on the "server side" of the TCP connection).</p>
|
||||
|
||||
<p>All Machi client access to the projection store SHOULD NOT use this
|
||||
module's API. Instead, clients should access indirectly via <a href="machi_cr_client.html"><code>machi_cr_client</code></a>, <a href="machi_proxy_flu1_client.html"><code>machi_proxy_flu1_client</code></a>, or <a href="machi_flu1_client.html"><code>machi_flu1_client</code></a>.</p>
|
||||
|
||||
The projection store is implemented by an Erlang/OTP <code>gen_server</code>
|
||||
process that is associated with each FLU. Conceptually, the
|
||||
projection store is an array of write-once registers. For each
|
||||
projection store register, the key is a 2-tuple of an epoch number
|
||||
(<code>non_neg_integer()</code> type) and a projection type (<code>public</code> or
|
||||
<code>private</code> type); the value is a projection data structure
|
||||
(<code>projection_v1()</code> type).
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-2">get_all_projections/2</a></td><td>Fetch all projection records of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-3">get_all_projections/3</a></td><td>Fetch all projection records of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-2">get_latest_epochid/2</a></td><td>Fetch the latest epoch number + checksum for type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-3">get_latest_epochid/3</a></td><td>Fetch the latest epoch number + checksum for type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_wedge_notify_pid-1">get_wedge_notify_pid/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-2">list_all_projections/2</a></td><td>Fetch all projection epoch numbers of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-3">list_all_projections/3</a></td><td>Fetch all projection epoch numbers of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read-3">read/3</a></td><td>Fetch the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</td></tr>
|
||||
<tr><td valign="top"><a href="#read-4">read/4</a></td><td>Fetch the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-2">read_latest_projection/2</a></td><td>Fetch the latest projection record for type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-3">read_latest_projection/3</a></td><td>Fetch the latest projection record for type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#set_consistency_mode-2">set_consistency_mode/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#set_wedge_notify_pid-2">set_wedge_notify_pid/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-3">start_link/3</a></td><td>Start a new projection store server.</td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#write-3">write/3</a></td><td>Write the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</td></tr>
|
||||
<tr><td valign="top"><a href="#write-4">write/4</a></td><td>Write the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-2">get_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Fetch all projection records of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-3">get_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch all projection records of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-2">get_latest_epochid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Fetch the latest epoch number + checksum for type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-3">get_latest_epochid/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the latest epoch number + checksum for type <code>ProjType</code>.
|
||||
projection.</p>
|
||||
|
||||
<h3 class="function"><a name="get_wedge_notify_pid-1">get_wedge_notify_pid/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_wedge_notify_pid(PidSpec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-2">list_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Fetch all projection epoch numbers of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-3">list_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch all projection epoch numbers of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read-3">read/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read(PidSpec, ProjType, Epoch) -> any()</tt></p>
|
||||
</div><p>Fetch the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</p>
|
||||
|
||||
<h3 class="function"><a name="read-4">read/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read(PidSpec, ProjType, Epoch, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-2">read_latest_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Fetch the latest projection record for type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-3">read_latest_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the latest projection record for type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="set_consistency_mode-2">set_consistency_mode/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>set_consistency_mode(PidSpec, CMode) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="set_wedge_notify_pid-2">set_wedge_notify_pid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>set_wedge_notify_pid(PidSpec, NotifyWedgeStateChanges) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-3">start_link/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(RegName, DataDir, NotifyWedgeStateChanges) -> any()</tt></p>
|
||||
</div><p><p>Start a new projection store server.</p>
|
||||
|
||||
The <code>DataDir</code> argument should be the same directory as specified
|
||||
for use by our companion FLU data server -- all file system paths
|
||||
used by this server are intended to be stored underneath a common
|
||||
file system parent directory as the FLU data server & sequencer
|
||||
servers.</p>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="write-3">write/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(PidSpec, ProjType, Proj) -> any()</tt></p>
|
||||
</div><p>Write the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</p>
|
||||
|
||||
<h3 class="function"><a name="write-4">write/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write(PidSpec, ProjType, Proj, Timeout) -> any()</tt></p>
|
||||
</div><p>Write the projection record type <code>ProjType</code> for epoch number <code>Epoch</code> .</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
323
edoc/machi_proxy_flu1_client.html
Normal file
323
edoc/machi_proxy_flu1_client.html
Normal file
|
|
@ -0,0 +1,323 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_proxy_flu1_client</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_proxy_flu1_client</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Erlang API for the Machi FLU TCP protocol version 1, with a
|
||||
proxy-process style API for hiding messy details such as TCP
|
||||
connection/disconnection with the remote Machi server.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Erlang API for the Machi FLU TCP protocol version 1, with a
|
||||
proxy-process style API for hiding messy details such as TCP
|
||||
connection/disconnection with the remote Machi server.</p>
|
||||
|
||||
<p>Machi is intentionally avoiding using distributed Erlang for
|
||||
Machi's communication. This design decision makes Erlang-side code
|
||||
more difficult & complex, but it's the price to pay for some
|
||||
language independence. Later in Machi's life cycle, we may (?) need to
|
||||
(re-)implement some components in a non-Erlang/BEAM-based language.</p>
|
||||
|
||||
<p>This module implements a "man in the middle" proxy between the
|
||||
Erlang client and Machi server (which is on the "far side" of a TCP
|
||||
connection to somewhere). This proxy process will always execute
|
||||
on the same Erlang node as the Erlang client that uses it. The
|
||||
proxy is intended to be a stable, long-lived process that survives
|
||||
TCP communication problems with the remote server.</p>
|
||||
|
||||
For a higher level interface, see <a href="machi_cr_client.html"><code>machi_cr_client</code></a>.
|
||||
For a lower level interface, see <a href="machi_flu1_client.html"><code>machi_flu1_client</code></a>.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append_chunk-4">append_chunk/4</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk-5">append_chunk/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-5">append_chunk_extra/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-6">append_chunk_extra/6</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-3">checksum_list/3</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-4">checksum_list/4</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#code_change-3">code_change/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-2">get_all_projections/2</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-3">get_all_projections/3</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_epoch_id-1">get_epoch_id/1</a></td><td>Get the <code>epoch_id()</code> of the FLU's current/latest projection.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_epoch_id-2">get_epoch_id/2</a></td><td>Get the <code>epoch_id()</code> of the FLU's current/latest projection.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-2">get_latest_epochid/2</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epochid-3">get_latest_epochid/3</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#handle_call-3">handle_call/3</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_cast-2">handle_cast/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#handle_info-2">handle_info/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#kick_projection_reaction-2">kick_projection_reaction/2</a></td><td>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</td></tr>
|
||||
<tr><td valign="top"><a href="#kick_projection_reaction-3">kick_projection_reaction/3</a></td><td>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-2">list_all_projections/2</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-3">list_all_projections/3</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-2">list_files/2</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-3">list_files/3</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#quit-1">quit/1</a></td><td>Quit & close the connection to remote FLU and stop our
|
||||
proxy process.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-6">read_chunk/6</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-7">read_chunk/7</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-2">read_latest_projection/2</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-3">read_latest_projection/3</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-3">read_projection/3</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-4">read_projection/4</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-1">start_link/1</a></td><td>Start a local, long-lived process that will be our steady
|
||||
& reliable communication proxy with the fickle & flaky
|
||||
remote Machi server.</td></tr>
|
||||
<tr><td valign="top"><a href="#start_proxies-1">start_proxies/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#stop_proxies-1">stop_proxies/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#terminate-2">terminate/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-5">trim_chunk/5</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#trim_chunk-6">trim_chunk/6</a></td><td>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-1">wedge_status/1</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-2">wedge_status/2</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-5">write_chunk/5</a></td><td>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-6">write_chunk/6</a></td><td>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-3">write_projection/3</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-4">write_projection/4</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-4">append_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec, EpochID, Prefix, Chunk) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-5">append_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(PidSpec, EpochID, Prefix, Chunk, Timeout) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-5">append_chunk_extra/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(PidSpec, EpochID, Prefix, Chunk, ChunkExtra) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-6">append_chunk_extra/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(PidSpec, EpochID, Prefix, Chunk, ChunkExtra, Timeout) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-3">checksum_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-4">checksum_list/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(PidSpec, EpochID, File, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="code_change-3">code_change/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>code_change(OldVsn, S, Extra) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-2">get_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-3">get_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_epoch_id-1">get_epoch_id/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_epoch_id(PidSpec) -> any()</tt></p>
|
||||
</div><p>Get the <code>epoch_id()</code> of the FLU's current/latest projection.</p>
|
||||
|
||||
<h3 class="function"><a name="get_epoch_id-2">get_epoch_id/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_epoch_id(PidSpec, Timeout) -> any()</tt></p>
|
||||
</div><p>Get the <code>epoch_id()</code> of the FLU's current/latest projection.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-2">get_latest_epochid/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epochid-3">get_latest_epochid/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epochid(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="handle_call-3">handle_call/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_call(Request, From, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_cast-2">handle_cast/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_cast(Msg, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="handle_info-2">handle_info/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>handle_info(Info, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="kick_projection_reaction-2">kick_projection_reaction/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>kick_projection_reaction(PidSpec, Options) -> any()</tt></p>
|
||||
</div><p>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</p>
|
||||
|
||||
<h3 class="function"><a name="kick_projection_reaction-3">kick_projection_reaction/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>kick_projection_reaction(PidSpec, Options, Timeout) -> any()</tt></p>
|
||||
</div><p>Kick (politely) the remote chain manager to react to a
|
||||
projection change.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-2">list_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-3">list_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-2">list_files/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec, EpochID) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-3">list_files/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(PidSpec, EpochID, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="quit-1">quit/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>quit(PidSpec) -> any()</tt></p>
|
||||
</div><p>Quit & close the connection to remote FLU and stop our
|
||||
proxy process.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-6">read_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec, EpochID, File, Offset, Size, Opts) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-7">read_chunk/7</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(PidSpec, EpochID, File, Offset, Size, Opts, Timeout) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-2">read_latest_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(PidSpec, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-3">read_latest_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(PidSpec, ProjType, Timeout) -> any()</tt></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-3">read_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(PidSpec, ProjType, Epoch) -> any()</tt></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-4">read_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(PidSpec, ProjType, Epoch, Timeout) -> any()</tt></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="start_link-1">start_link/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link(P_srvr) -> any()</tt></p>
|
||||
</div><p>Start a local, long-lived process that will be our steady
|
||||
& reliable communication proxy with the fickle & flaky
|
||||
remote Machi server.</p>
|
||||
|
||||
<h3 class="function"><a name="start_proxies-1">start_proxies/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_proxies(MembersDict) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="stop_proxies-1">stop_proxies/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>stop_proxies(ProxiesDict) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="terminate-2">terminate/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>terminate(Reason, S) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-5">trim_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec, EpochID, File, Offset, Size) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="trim_chunk-6">trim_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trim_chunk(PidSpec, EpochID, File, Offset, Chunk, Timeout) -> any()</tt></p>
|
||||
</div><p>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-1">wedge_status/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(PidSpec) -> any()</tt></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-2">wedge_status/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(PidSpec, Timeout) -> any()</tt></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-5">write_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec, EpochID, File, Offset, Chunk) -> any()</tt></p>
|
||||
</div><p>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-6">write_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(PidSpec, EpochID, File, Offset, Chunk, Timeout) -> any()</tt></p>
|
||||
</div><p>Write a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-3">write_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(PidSpec, ProjType, Proj) -> any()</tt></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-4">write_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(PidSpec, ProjType, Proj, Timeout) -> any()</tt></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
23
edoc/machi_sequencer.html
Normal file
23
edoc/machi_sequencer.html
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_sequencer</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_sequencer</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li></ul>"Mothballed" sequencer code, perhaps to be reused sometime in
|
||||
the future?.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>"Mothballed" sequencer code, perhaps to be reused sometime in
|
||||
the future?<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Jul 1 2015, 18:32:19.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
42
edoc/machi_sup.html
Normal file
42
edoc/machi_sup.html
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_sup</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_sup</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Top Machi application supervisor.
|
||||
|
||||
<p><b>Behaviours:</b> <a href="supervisor.html"><tt>supervisor</tt></a>.</p>
|
||||
|
||||
<h2><a name="description">Description</a></h2><p>Top Machi application supervisor.</p>
|
||||
|
||||
See <a href="machi_flu_psup.html"><code>machi_flu_psup</code></a> for an illustration of the entire Machi
|
||||
application process structure.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#init-1">init/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#start_link-0">start_link/0</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="init-1">init/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>init(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="start_link-0">start_link/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>start_link() -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
288
edoc/machi_util.html
Normal file
288
edoc/machi_util.html
Normal file
|
|
@ -0,0 +1,288 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_util</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_util</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>Miscellaneous utility functions.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>Miscellaneous utility functions.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#bin_to_hexstr-1">bin_to_hexstr/1</a></td><td>Convert a <code>binary()</code> to a hexadecimal string.</td></tr>
|
||||
<tr><td valign="top"><a href="#bool2int-1">bool2int/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_chunk-1">checksum_chunk/1</a></td><td>Calculate a checksum for a chunk of file data.</td></tr>
|
||||
<tr><td valign="top"><a href="#combinations-1">combinations/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#connect-2">connect/2</a></td><td>Create a TCP connection to a remote Machi server.</td></tr>
|
||||
<tr><td valign="top"><a href="#connect-3">connect/3</a></td><td>Create a TCP connection to a remote Machi server.</td></tr>
|
||||
<tr><td valign="top"><a href="#hexstr_to_bin-1">hexstr_to_bin/1</a></td><td>Convert a hexadecimal string to a <code>binary()</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#hexstr_to_int-1">hexstr_to_int/1</a></td><td>Convert a hexadecimal string to an integer.</td></tr>
|
||||
<tr><td valign="top"><a href="#increment_max_filenum-2">increment_max_filenum/2</a></td><td>Increase the file size of a config file, which is used as the
|
||||
basis for a minimum sequence number.</td></tr>
|
||||
<tr><td valign="top"><a href="#info_msg-2">info_msg/2</a></td><td>Log an 'info' level message.</td></tr>
|
||||
<tr><td valign="top"><a href="#int2bool-1">int2bool/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#int_to_hexbin-2">int_to_hexbin/2</a></td><td>Convert an integer into a hexadecimal string (in <code>binary()</code>
|
||||
form) whose length is based on <code>I_size</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#int_to_hexstr-2">int_to_hexstr/2</a></td><td>Convert an integer into a hexadecimal string whose length is
|
||||
based on <code>I_size</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#is_valid_filename-1">is_valid_filename/1</a></td><td>Given a filename, return true if it is a valid machi filename,
|
||||
false otherwise.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_binary-1">make_binary/1</a></td><td>Convert a compatible Erlang data type into a <code>binary()</code> equivalent.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_checksum_filename-2">make_checksum_filename/2</a></td><td>Calculate a checksum file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_checksum_filename-4">make_checksum_filename/4</a></td><td>Calculate a checksum file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_config_filename-2">make_config_filename/2</a></td><td>Calculate a config file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_data_filename-2">make_data_filename/2</a></td><td>Calculate a file data file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_data_filename-4">make_data_filename/4</a></td><td>Calculate a file data file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_projection_filename-2">make_projection_filename/2</a></td><td>Calculate a projection store file path, by common convention.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_regname-1">make_regname/1</a></td><td>Create a registered name atom for FLU sequencer internal
|
||||
rendezvous/message passing use.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_string-1">make_string/1</a></td><td>Convert a compatible Erlang data type into a <code>string()</code> equivalent.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_tagged_csum-1">make_tagged_csum/1</a></td><td>Create a tagged checksum.</td></tr>
|
||||
<tr><td valign="top"><a href="#make_tagged_csum-2">make_tagged_csum/2</a></td><td>Makes tagged csum.</td></tr>
|
||||
<tr><td valign="top"><a href="#mbytes-1">mbytes/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#mk_order-2">mk_order/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#ordered_combinations-1">ordered_combinations/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#parse_filename-1">parse_filename/1</a></td><td>Given a machi filename, return a set of components in a list.</td></tr>
|
||||
<tr><td valign="top"><a href="#perms-1">perms/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#permutations-1">permutations/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#pretty_time-0">pretty_time/0</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#pretty_time-2">pretty_time/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#read_max_filenum-2">read_max_filenum/2</a></td><td>Read the file size of a config file, which is used as the
|
||||
basis for a minimum sequence number.</td></tr>
|
||||
<tr><td valign="top"><a href="#unmake_tagged_csum-1">unmake_tagged_csum/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#verb-1">verb/1</a></td><td>Log a verbose message.</td></tr>
|
||||
<tr><td valign="top"><a href="#verb-2">verb/2</a></td><td>Log a verbose message.</td></tr>
|
||||
<tr><td valign="top"><a href="#wait_for_death-2">wait_for_death/2</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#wait_for_life-2">wait_for_life/2</a></td><td></td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="bin_to_hexstr-1">bin_to_hexstr/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>bin_to_hexstr(X1::binary()) -> string()</tt><br></p>
|
||||
</div><p>Convert a <code>binary()</code> to a hexadecimal string.</p>
|
||||
|
||||
<h3 class="function"><a name="bool2int-1">bool2int/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>bool2int(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="checksum_chunk-1">checksum_chunk/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_chunk(Chunk::binary() | iolist()) -> binary()</tt><br></p>
|
||||
</div><p>Calculate a checksum for a chunk of file data.</p>
|
||||
|
||||
<h3 class="function"><a name="combinations-1">combinations/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>combinations(L) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="connect-2">connect/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connect(Host::<a href="inet.html#type-ip_address">inet:ip_address()</a> | <a href="inet.html#type-hostname">inet:hostname()</a>, Port::<a href="inet.html#type-port_number">inet:port_number()</a>) -> port()</tt><br></p>
|
||||
</div><p>Create a TCP connection to a remote Machi server.</p>
|
||||
|
||||
<h3 class="function"><a name="connect-3">connect/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connect(Host::<a href="inet.html#type-ip_address">inet:ip_address()</a> | <a href="inet.html#type-hostname">inet:hostname()</a>, Port::<a href="inet.html#type-port_number">inet:port_number()</a>, Timeout::timeout()) -> port()</tt><br></p>
|
||||
</div><p>Create a TCP connection to a remote Machi server.</p>
|
||||
|
||||
<h3 class="function"><a name="hexstr_to_bin-1">hexstr_to_bin/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>hexstr_to_bin(S::string() | binary()) -> binary()</tt><br></p>
|
||||
</div><p>Convert a hexadecimal string to a <code>binary()</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="hexstr_to_int-1">hexstr_to_int/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>hexstr_to_int(X::string() | binary()) -> non_neg_integer()</tt><br></p>
|
||||
</div><p>Convert a hexadecimal string to an integer.</p>
|
||||
|
||||
<h3 class="function"><a name="increment_max_filenum-2">increment_max_filenum/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>increment_max_filenum(DataDir::string(), Prefix::string()) -> ok | {error, term()}</tt><br></p>
|
||||
</div><p>Increase the file size of a config file, which is used as the
|
||||
basis for a minimum sequence number.</p>
|
||||
|
||||
<h3 class="function"><a name="info_msg-2">info_msg/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>info_msg(Fmt::string(), Args::list()) -> term()</tt><br></p>
|
||||
</div><p>Log an 'info' level message.</p>
|
||||
|
||||
<h3 class="function"><a name="int2bool-1">int2bool/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>int2bool(I) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="int_to_hexbin-2">int_to_hexbin/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>int_to_hexbin(I::non_neg_integer(), I_size::non_neg_integer()) -> binary()</tt><br></p>
|
||||
</div><p>Convert an integer into a hexadecimal string (in <code>binary()</code>
|
||||
form) whose length is based on <code>I_size</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="int_to_hexstr-2">int_to_hexstr/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>int_to_hexstr(I::non_neg_integer(), I_size::non_neg_integer()) -> string()</tt><br></p>
|
||||
</div><p>Convert an integer into a hexadecimal string whose length is
|
||||
based on <code>I_size</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="is_valid_filename-1">is_valid_filename/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>is_valid_filename(Filename::string()) -> true | false</tt><br></p>
|
||||
</div><p>Given a filename, return true if it is a valid machi filename,
|
||||
false otherwise.</p>
|
||||
|
||||
<h3 class="function"><a name="make_binary-1">make_binary/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_binary(X::binary() | iolist()) -> binary()</tt><br></p>
|
||||
</div><p>Convert a compatible Erlang data type into a <code>binary()</code> equivalent.</p>
|
||||
|
||||
<h3 class="function"><a name="make_checksum_filename-2">make_checksum_filename/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_checksum_filename(DataDir::string(), FileName::[] | string() | binary()) -> string()</tt><br></p>
|
||||
</div><p>Calculate a checksum file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_checksum_filename-4">make_checksum_filename/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_checksum_filename(DataDir::string(), Prefix::string(), SequencerName::atom() | string() | binary(), FileNum::integer()) -> string()</tt><br></p>
|
||||
</div><p>Calculate a checksum file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_config_filename-2">make_config_filename/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_config_filename(DataDir::string(), Prefix::string()) -> string()</tt><br></p>
|
||||
</div><p>Calculate a config file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_data_filename-2">make_data_filename/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_data_filename(DataDir::string(), File::[] | string() | binary()) -> {binary(), string()}</tt><br></p>
|
||||
</div><p>Calculate a file data file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_data_filename-4">make_data_filename/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_data_filename(DataDir::string(), Prefix::string(), SequencerName::atom() | string() | binary(), FileNum::integer() | string()) -> {binary(), string()}</tt><br></p>
|
||||
</div><p>Calculate a file data file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_projection_filename-2">make_projection_filename/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_projection_filename(DataDir::string(), File::[] | string()) -> string()</tt><br></p>
|
||||
</div><p>Calculate a projection store file path, by common convention.</p>
|
||||
|
||||
<h3 class="function"><a name="make_regname-1">make_regname/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_regname(Prefix::binary() | string()) -> atom()</tt><br></p>
|
||||
</div><p>Create a registered name atom for FLU sequencer internal
|
||||
rendezvous/message passing use.</p>
|
||||
|
||||
<h3 class="function"><a name="make_string-1">make_string/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_string(X::binary() | iolist()) -> string()</tt><br></p>
|
||||
</div><p>Convert a compatible Erlang data type into a <code>string()</code> equivalent.</p>
|
||||
|
||||
<h3 class="function"><a name="make_tagged_csum-1">make_tagged_csum/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_tagged_csum(X1) -> any()</tt></p>
|
||||
</div><p>Create a tagged checksum</p>
|
||||
|
||||
<h3 class="function"><a name="make_tagged_csum-2">make_tagged_csum/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>make_tagged_csum(X1::<a href="machi_dt.html#type-csum_tag">machi_dt:csum_tag()</a>, SHA::binary()) -> <a href="machi_dt.html#type-chunk_csum">machi_dt:chunk_csum()</a></tt><br></p>
|
||||
</div><p>Makes tagged csum. Each meanings are:
|
||||
none / ?CSUM_TAG_NONE
|
||||
- a suspicious and nonsense checksum
|
||||
client_sha / ?CSUM_TAG_CLIENT_SHA
|
||||
- a valid checksum given by client and stored in server
|
||||
server_sha / ?CSUM_TAG_SERVER_SHA
|
||||
- a valid checksum generated by and stored in server
|
||||
server_regen_sha / ?CSUM_TAG_SERVER_REGEN_SHA
|
||||
- a valid checksum generated by server in an ad hoc manner, not stored in server</p>
|
||||
|
||||
<h3 class="function"><a name="mbytes-1">mbytes/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>mbytes(Size) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="mk_order-2">mk_order/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>mk_order(UPI2, Repair1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="ordered_combinations-1">ordered_combinations/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>ordered_combinations(Master) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="parse_filename-1">parse_filename/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>parse_filename(Filename::string()) -> [string()]</tt><br></p>
|
||||
</div><p>Given a machi filename, return a set of components in a list.
|
||||
The components will be:
|
||||
<ul>
|
||||
<li>Prefix</li>
|
||||
<li>UUID</li>
|
||||
<li>Sequence number</li>
|
||||
</ul>
|
||||
|
||||
Invalid filenames will return an empty list.</p>
|
||||
|
||||
<h3 class="function"><a name="perms-1">perms/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>perms(L) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="permutations-1">permutations/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>permutations(L) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="pretty_time-0">pretty_time/0</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>pretty_time() -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="pretty_time-2">pretty_time/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>pretty_time(X1, MSec) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="read_max_filenum-2">read_max_filenum/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_max_filenum(DataDir::string(), Prefix::string()) -> non_neg_integer()</tt><br></p>
|
||||
</div><p>Read the file size of a config file, which is used as the
|
||||
basis for a minimum sequence number.</p>
|
||||
|
||||
<h3 class="function"><a name="unmake_tagged_csum-1">unmake_tagged_csum/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>unmake_tagged_csum(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="verb-1">verb/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verb(Fmt::string()) -> ok</tt><br></p>
|
||||
</div><p>Log a verbose message.</p>
|
||||
|
||||
<h3 class="function"><a name="verb-2">verb/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>verb(Fmt::string(), Args::list()) -> ok</tt><br></p>
|
||||
</div><p>Log a verbose message.</p>
|
||||
|
||||
<h3 class="function"><a name="wait_for_death-2">wait_for_death/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wait_for_death(Pid, Iters) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="wait_for_life-2">wait_for_life/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wait_for_life(Reg, Iters) -> any()</tt></p>
|
||||
</div>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
260
edoc/machi_yessir_client.html
Normal file
260
edoc/machi_yessir_client.html
Normal file
|
|
@ -0,0 +1,260 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Module machi_yessir_client</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<hr>
|
||||
|
||||
<h1>Module machi_yessir_client</h1>
|
||||
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>"Yes, sir!" style dummy/mock client facade.
|
||||
|
||||
|
||||
<h2><a name="description">Description</a></h2>"Yes, sir!" style dummy/mock client facade.
|
||||
<h2><a name="index">Function Index</a></h2>
|
||||
<table width="100%" border="1" cellspacing="0" cellpadding="2" summary="function index"><tr><td valign="top"><a href="#append_chunk-4">append_chunk/4</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk-5">append_chunk/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-5">append_chunk_extra/5</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</td></tr>
|
||||
<tr><td valign="top"><a href="#append_chunk_extra-6">append_chunk_extra/6</a></td><td>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-3">checksum_list/3</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#checksum_list-4">checksum_list/4</a></td><td>Fetch the list of chunk checksums for <code>File</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#connect-1">connect/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#connected_p-1">connected_p/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#delete_migration-3">delete_migration/3</a></td><td>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</td></tr>
|
||||
<tr><td valign="top"><a href="#delete_migration-4">delete_migration/4</a></td><td>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</td></tr>
|
||||
<tr><td valign="top"><a href="#disconnect-1">disconnect/1</a></td><td></td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-2">get_all_projections/2</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_all_projections-3">get_all_projections/3</a></td><td>Get all projections from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epoch-2">get_latest_epoch/2</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#get_latest_epoch-3">get_latest_epoch/3</a></td><td>Get the latest epoch number + checksum from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-2">list_all_projections/2</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_all_projections-3">list_all_projections/3</a></td><td>Get all epoch numbers from the FLU's projection store.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-2">list_files/2</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#list_files-3">list_files/3</a></td><td>Fetch the list of all files on the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#quit-1">quit/1</a></td><td>Quit & close the connection to remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-5">read_chunk/5</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_chunk-6">read_chunk/6</a></td><td>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-2">read_latest_projection/2</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_latest_projection-3">read_latest_projection/3</a></td><td>Get the latest projection from the FLU's projection store for <code>ProjType</code></td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-3">read_projection/3</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#read_projection-4">read_projection/4</a></td><td>Read a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#trunc_hack-3">trunc_hack/3</a></td><td>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</td></tr>
|
||||
<tr><td valign="top"><a href="#trunc_hack-4">trunc_hack/4</a></td><td>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-1">wedge_status/1</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#wedge_status-2">wedge_status/2</a></td><td>Fetch the wedge status from the remote FLU.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-5">write_chunk/5</a></td><td>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_chunk-6">write_chunk/6</a></td><td>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-3">write_projection/3</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
<tr><td valign="top"><a href="#write_projection-4">write_projection/4</a></td><td>Write a projection <code>Proj</code> of type <code>ProjType</code>.</td></tr>
|
||||
</table>
|
||||
|
||||
<h2><a name="functions">Function Details</a></h2>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-4">append_chunk/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(Sock, EpochID, Prefix, Chunk) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk-5">append_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk(Host, TcpPort, EpochID, Prefix, Chunk) -> any()</tt></p>
|
||||
</div><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-5">append_chunk_extra/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(Yessir, EpochID, Prefix, Chunk, ChunkExtra) -> any()</tt></p>
|
||||
</div><p><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</p>
|
||||
|
||||
For example, if the <code>Chunk</code> size is 1 KByte and <code>Extra</code> is 4K Bytes, then
|
||||
the file offsets that follow <code>Chunk</code>'s position for the following 4K will
|
||||
be reserved by the file sequencer for later write(s) by the
|
||||
<code>write_chunk()</code> API.</p>
|
||||
|
||||
<h3 class="function"><a name="append_chunk_extra-6">append_chunk_extra/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>append_chunk_extra(Host, TcpPort, EpochID, Prefix, Chunk, ChunkExtra) -> any()</tt></p>
|
||||
</div><p><p>Append a chunk (binary- or iolist-style) of data to a file
|
||||
with <code>Prefix</code> and also request an additional <code>Extra</code> bytes.</p>
|
||||
|
||||
For example, if the <code>Chunk</code> size is 1 KByte and <code>Extra</code> is 4K Bytes, then
|
||||
the file offsets that follow <code>Chunk</code>'s position for the following 4K will
|
||||
be reserved by the file sequencer for later write(s) by the
|
||||
<code>write_chunk()</code> API.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-3">checksum_list/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(Yessir, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="checksum_list-4">checksum_list/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>checksum_list(Host, TcpPort, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Fetch the list of chunk checksums for <code>File</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="connect-1">connect/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connect(P_srvr) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="connected_p-1">connected_p/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>connected_p(X1) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="delete_migration-3">delete_migration/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete_migration(Yessir, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</p>
|
||||
|
||||
<h3 class="function"><a name="delete_migration-4">delete_migration/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>delete_migration(Host, TcpPort, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Restricted API: Delete a file after it has been successfully
|
||||
migrated.</p>
|
||||
|
||||
<h3 class="function"><a name="disconnect-1">disconnect/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>disconnect(Yessir) -> any()</tt></p>
|
||||
</div>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-2">get_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(Yessir, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_all_projections-3">get_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_all_projections(Host, TcpPort, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all projections from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epoch-2">get_latest_epoch/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epoch(Sock, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="get_latest_epoch-3">get_latest_epoch/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>get_latest_epoch(Host, TcpPort, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest epoch number + checksum from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-2">list_all_projections/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(Sock, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_all_projections-3">list_all_projections/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_all_projections(Host, TcpPort, ProjType) -> any()</tt></p>
|
||||
</div><p>Get all epoch numbers from the FLU's projection store.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-2">list_files/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(Yessir, EpochID) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="list_files-3">list_files/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>list_files(Host, TcpPort, EpochID) -> any()</tt></p>
|
||||
</div><p>Fetch the list of all files on the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="quit-1">quit/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>quit(X1) -> any()</tt></p>
|
||||
</div><p>Quit & close the connection to remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-5">read_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(Yessir, EpochID, File, Offset, Size) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_chunk-6">read_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_chunk(Host, TcpPort, EpochID, File, Offset, Size) -> any()</tt></p>
|
||||
</div><p>Read a chunk of data of size <code>Size</code> from <code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-2">read_latest_projection/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(Yessir, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_latest_projection-3">read_latest_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_latest_projection(Host, TcpPort, ProjType) -> any()</tt></p>
|
||||
</div><p>Get the latest projection from the FLU's projection store for <code>ProjType</code></p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-3">read_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(Yessir, ProjType, Epoch) -> any()</tt></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="read_projection-4">read_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>read_projection(Host, TcpPort, ProjType, Epoch) -> any()</tt></p>
|
||||
</div><p>Read a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="trunc_hack-3">trunc_hack/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trunc_hack(Yessir, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</p>
|
||||
|
||||
<h3 class="function"><a name="trunc_hack-4">trunc_hack/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>trunc_hack(Host, TcpPort, EpochID, File) -> any()</tt></p>
|
||||
</div><p>Restricted API: Truncate a file after it has been successfully
|
||||
erasure coded.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-1">wedge_status/1</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(Sock) -> any()</tt></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="wedge_status-2">wedge_status/2</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>wedge_status(Host, TcpPort) -> any()</tt></p>
|
||||
</div><p>Fetch the wedge status from the remote FLU.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-5">write_chunk/5</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(Yessir, EpochID, File, Offset, Chunk) -> any()</tt></p>
|
||||
</div><p>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_chunk-6">write_chunk/6</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_chunk(Host, TcpPort, EpochID, File, Offset, Chunk) -> any()</tt></p>
|
||||
</div><p>Restricted API: Write a chunk of already-sequenced data to
|
||||
<code>File</code> at <code>Offset</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-3">write_projection/3</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(Yessir, ProjType, Proj) -> any()</tt></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
|
||||
<h3 class="function"><a name="write_projection-4">write_projection/4</a></h3>
|
||||
<div class="spec">
|
||||
<p><tt>write_projection(Host, TcpPort, ProjType, Proj) -> any()</tt></p>
|
||||
</div><p>Write a projection <code>Proj</code> of type <code>ProjType</code>.</p>
|
||||
<hr>
|
||||
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
43
edoc/modules-frame.html
Normal file
43
edoc/modules-frame.html
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<title>The machi application</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<h2 class="indextitle">Modules</h2>
|
||||
<table width="100%" border="0" summary="list of modules">
|
||||
<tr><td><a href="machi_admin_util.html" target="overviewFrame" class="module">machi_admin_util</a></td></tr>
|
||||
<tr><td><a href="machi_app.html" target="overviewFrame" class="module">machi_app</a></td></tr>
|
||||
<tr><td><a href="machi_basho_bench_driver.html" target="overviewFrame" class="module">machi_basho_bench_driver</a></td></tr>
|
||||
<tr><td><a href="machi_chain_manager1.html" target="overviewFrame" class="module">machi_chain_manager1</a></td></tr>
|
||||
<tr><td><a href="machi_chain_repair.html" target="overviewFrame" class="module">machi_chain_repair</a></td></tr>
|
||||
<tr><td><a href="machi_chash.html" target="overviewFrame" class="module">machi_chash</a></td></tr>
|
||||
<tr><td><a href="machi_cinfo.html" target="overviewFrame" class="module">machi_cinfo</a></td></tr>
|
||||
<tr><td><a href="machi_config.html" target="overviewFrame" class="module">machi_config</a></td></tr>
|
||||
<tr><td><a href="machi_cr_client.html" target="overviewFrame" class="module">machi_cr_client</a></td></tr>
|
||||
<tr><td><a href="machi_csum_table.html" target="overviewFrame" class="module">machi_csum_table</a></td></tr>
|
||||
<tr><td><a href="machi_dt.html" target="overviewFrame" class="module">machi_dt</a></td></tr>
|
||||
<tr><td><a href="machi_file_proxy.html" target="overviewFrame" class="module">machi_file_proxy</a></td></tr>
|
||||
<tr><td><a href="machi_file_proxy_sup.html" target="overviewFrame" class="module">machi_file_proxy_sup</a></td></tr>
|
||||
<tr><td><a href="machi_fitness.html" target="overviewFrame" class="module">machi_fitness</a></td></tr>
|
||||
<tr><td><a href="machi_flu1.html" target="overviewFrame" class="module">machi_flu1</a></td></tr>
|
||||
<tr><td><a href="machi_flu1_client.html" target="overviewFrame" class="module">machi_flu1_client</a></td></tr>
|
||||
<tr><td><a href="machi_flu_filename_mgr.html" target="overviewFrame" class="module">machi_flu_filename_mgr</a></td></tr>
|
||||
<tr><td><a href="machi_flu_metadata_mgr.html" target="overviewFrame" class="module">machi_flu_metadata_mgr</a></td></tr>
|
||||
<tr><td><a href="machi_flu_metadata_mgr_sup.html" target="overviewFrame" class="module">machi_flu_metadata_mgr_sup</a></td></tr>
|
||||
<tr><td><a href="machi_flu_psup.html" target="overviewFrame" class="module">machi_flu_psup</a></td></tr>
|
||||
<tr><td><a href="machi_flu_sup.html" target="overviewFrame" class="module">machi_flu_sup</a></td></tr>
|
||||
<tr><td><a href="machi_lifecycle_mgr.html" target="overviewFrame" class="module">machi_lifecycle_mgr</a></td></tr>
|
||||
<tr><td><a href="machi_merkle_tree.html" target="overviewFrame" class="module">machi_merkle_tree</a></td></tr>
|
||||
<tr><td><a href="machi_pb_high_client.html" target="overviewFrame" class="module">machi_pb_high_client</a></td></tr>
|
||||
<tr><td><a href="machi_pb_translate.html" target="overviewFrame" class="module">machi_pb_translate</a></td></tr>
|
||||
<tr><td><a href="machi_plist.html" target="overviewFrame" class="module">machi_plist</a></td></tr>
|
||||
<tr><td><a href="machi_projection.html" target="overviewFrame" class="module">machi_projection</a></td></tr>
|
||||
<tr><td><a href="machi_projection_store.html" target="overviewFrame" class="module">machi_projection_store</a></td></tr>
|
||||
<tr><td><a href="machi_proxy_flu1_client.html" target="overviewFrame" class="module">machi_proxy_flu1_client</a></td></tr>
|
||||
<tr><td><a href="machi_sup.html" target="overviewFrame" class="module">machi_sup</a></td></tr>
|
||||
<tr><td><a href="machi_util.html" target="overviewFrame" class="module">machi_util</a></td></tr>
|
||||
<tr><td><a href="machi_yessir_client.html" target="overviewFrame" class="module">machi_yessir_client</a></td></tr></table>
|
||||
</body>
|
||||
</html>
|
||||
185
edoc/overview-summary.html
Normal file
185
edoc/overview-summary.html
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
||||
<title>Machi: a small village of replicated files
|
||||
</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<div class="navbar"><a name="#navbar_top"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<h1>Machi: a small village of replicated files
|
||||
</h1>
|
||||
|
||||
|
||||
<h3><a name="About_This_EDoc_Documentation">About This EDoc Documentation</a></h3>
|
||||
|
||||
<p>This EDoc-style documentation will concern itself only with Erlang
|
||||
function APIs and function & data types. Higher-level design and
|
||||
commentary will remain outside of the Erlang EDoc system; please see
|
||||
the "Pointers to Other Machi Documentation" section below for more
|
||||
details.</p>
|
||||
|
||||
<p>Readers should beware that this documentation may be out-of-sync with
|
||||
the source code. When in doubt, use the <code>make edoc</code> command to
|
||||
regenerate all HTML pages.</p>
|
||||
|
||||
<p>It is the developer's responsibility to re-generate the documentation
|
||||
periodically and commit it to the Git repo.</p>
|
||||
|
||||
<h3><a name="Machi_Code_Overview">Machi Code Overview</a></h3>
|
||||
|
||||
<h4><a name="Chain_Manager">Chain Manager</a></h4>
|
||||
|
||||
<p>The Chain Manager is responsible for managing the state of Machi's
|
||||
"Chain Replication" state. This role is roughly analogous to the
|
||||
"Riak Core" application inside of Riak, which takes care of
|
||||
coordinating replica placement and replica repair.</p>
|
||||
|
||||
<p>For each primitive data server in the cluster, a Machi FLU, there is a
|
||||
Chain Manager process that manages its FLU's role within the Machi
|
||||
cluster's Chain Replication scheme. Each Chain Manager process
|
||||
executes locally and independently to manage the distributed state of
|
||||
a single Machi Chain Replication chain.</p>
|
||||
|
||||
<ul>
|
||||
|
||||
<li><p> To contrast with Riak Core ... Riak Core's claimant process is
|
||||
solely responsible for managing certain critical aspects of
|
||||
Riak Core distributed state. Machi's Chain Manager process
|
||||
performs similar tasks as Riak Core's claimant. However, Machi
|
||||
has several active Chain Manager processes, one per FLU server,
|
||||
instead of a single active process like Core's claimant. Each
|
||||
Chain Manager process acts independently; each is constrained
|
||||
so that it will reach consensus via independent computation
|
||||
& action.</p>
|
||||
|
||||
Full discussion of this distributed consensus is outside the
|
||||
scope of this document; see the "Pointers to Other Machi
|
||||
Documentation" section below for more information.
|
||||
</li>
|
||||
<li> Machi differs from a Riak Core application because Machi's
|
||||
replica placement policy is simply, "All Machi servers store
|
||||
replicas of all Machi files".
|
||||
Machi is intended to be a primitive building block for creating larger
|
||||
cluster-of-clusters where files are
|
||||
distributed/fragmented/sharded across a large pool of
|
||||
independent Machi clusters.
|
||||
</li>
|
||||
<li> See
|
||||
<a href="https://www.usenix.org/legacy/events/osdi04/tech/renesse.html" target="_top"><tt>https://www.usenix.org/legacy/events/osdi04/tech/renesse.html</tt></a>
|
||||
for a copy of the paper, "Chain Replication for Supporting High
|
||||
Throughput and Availability" by Robbert van Renesse and Fred
|
||||
B. Schneider.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<h4><a name="FLU">FLU</a></h4>
|
||||
|
||||
<p>The FLU is the basic storage server for Machi.</p>
|
||||
|
||||
<ul>
|
||||
<li> The name FLU is taken from "flash storage unit" from the paper
|
||||
"CORFU: A Shared Log Design for Flash Clusters" by
|
||||
Balakrishnan, Malkhi, Prabhakaran, and Wobber. See
|
||||
<a href="https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/balakrishnan" target="_top"><tt>https://www.usenix.org/conference/nsdi12/technical-sessions/presentation/balakrishnan</tt></a>
|
||||
</li>
|
||||
<li> In CORFU, the sequencer step is a prerequisite step that is
|
||||
performed by a separate component, the Sequencer.
|
||||
In Machi, the <code>append_chunk()</code> protocol message has
|
||||
an implicit "sequencer" operation applied by the "head" of the
|
||||
Machi Chain Replication chain. If a client wishes to write
|
||||
data that has already been assigned a sequencer position, then
|
||||
the <code>write_chunk()</code> API function is used.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p>For each FLU, there are three independent tasks that are implemented
|
||||
using three different Erlang processes:</p>
|
||||
|
||||
<ul>
|
||||
<li> A FLU server, implemented primarily by <code>machi_flu.erl</code>.
|
||||
</li>
|
||||
<li> A projection store server, implemented primarily by
|
||||
<code>machi_projection_store.erl</code>.
|
||||
</li>
|
||||
<li> A chain state manager server, implemented primarily by
|
||||
<code>machi_chain_manager1.erl</code>.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p>From the perspective of failure detection, it is very convenient that
|
||||
all three FLU-related services (file server, sequencer server, and
|
||||
projection server) are accessed using the same single TCP port.</p>
|
||||
|
||||
<h4><a name="Projection_(data_structure)">Projection (data structure)</a></h4>
|
||||
|
||||
<p>The projection is a data structure that specifies the current state
|
||||
of the Machi cluster: all FLUs, which FLUS are considered
|
||||
up/running or down/crashed/stopped, which FLUs are actively
|
||||
participants in the Chain Replication protocol, and which FLUs are
|
||||
under "repair" (i.e., having their data resyncronized when
|
||||
newly-added to a cluster or when restarting after a crash).</p>
|
||||
|
||||
<h4><a name="Projection_Store_(server)">Projection Store (server)</a></h4>
|
||||
|
||||
<p>The projection store is a storage service that is implemented by an
|
||||
Erlang/OTP <code>gen_server</code> process that is associated with each
|
||||
FLU. Conceptually, the projection store is an array of
|
||||
write-once registers. For each projection store register, the
|
||||
key is a 2-tuple of an epoch number (<code>non_neg_integer()</code> type)
|
||||
and a projection type (<code>public</code> or <code>private</code> type); the value is
|
||||
a projection data structure (<code>projection_v1()</code> type).</p>
|
||||
|
||||
<h4><a name="Client_and_Proxy_Client">Client and Proxy Client</a></h4>
|
||||
|
||||
<p>Machi is intentionally avoiding using distributed Erlang for Machi's
|
||||
communication. This design decision makes Erlang-side code more
|
||||
difficult & complex but allows us the freedom of implementing
|
||||
parts of Machi in other languages without major
|
||||
protocol&API&glue code changes later in the product's
|
||||
lifetime.</p>
|
||||
|
||||
<p>There are two layers of interface for Machi clients.</p>
|
||||
|
||||
<ul>
|
||||
<li> The <code>machi_flu1_client</code> module implements an API that uses a
|
||||
TCP socket directly.
|
||||
</li>
|
||||
<li> The <code>machi_proxy_flu1_client</code> module implements an API that
|
||||
uses a local, long-lived <code>gen_server</code> process as a proxy for
|
||||
the remote, perhaps disconnected-or-crashed Machi FLU server.
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p>The types for both modules ought to be the same. However, due to
|
||||
rapid code churn, some differences might exist. Any major difference
|
||||
is (almost by definition) a bug: please open a GitHub issue to request
|
||||
a correction.</p>
|
||||
|
||||
<h3><a name="TODO_notes">TODO notes</a></h3>
|
||||
|
||||
<p>Any use of the string "TODO" in upper/lower/mixed case, anywhere in
|
||||
the code, is a reminder signal of unfinished work.</p>
|
||||
|
||||
<h3><a name="Pointers_to_Other_Machi_Documentation">Pointers to Other Machi Documentation</a></h3>
|
||||
|
||||
<ul>
|
||||
<li> If you are viewing this document locally, please look in the
|
||||
<code>../doc/</code> directory,
|
||||
</li>
|
||||
<li> If you are viewing this document via the Web, please find the
|
||||
documentation via this link:
|
||||
<a href="http://github.com/basho/machi/tree/master/doc/" target="_top"><tt>http://github.com/basho/machi/tree/master/doc/</tt></a>
|
||||
Please be aware that this link points to the <code>master</code> branch
|
||||
of the Machi source repository and therefore may be
|
||||
out-of-sync with non-<code>master</code> branch code.
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
|
||||
<hr>
|
||||
<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
|
||||
<p><i>Generated by EDoc, Dec 8 2015, 21:37:38.</i></p>
|
||||
</body>
|
||||
</html>
|
||||
14
edoc/overview.edoc
Normal file
14
edoc/overview.edoc
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
|
||||
@title Machi: a small village of replicated files
|
||||
|
||||
@doc
|
||||
|
||||
Documentation for Machi is an ongoing challenge. Much of the
|
||||
high-level design & commentary are outside of the Erlang EDoc system
|
||||
|
||||
Zoom2 zoom zoom zoom boom boom boom boom
|
||||
|
||||
Rumba tango Rumba tango Rumba tango Rumba tango Rumba tango Rumba
|
||||
tango Rumba tango Rumba tango Rumba tango Rumba tango Rumba tango
|
||||
Rumba tango Rumba tango Rumba tango Rumba tango Rumba tango Rumba
|
||||
tango Rumba tango Rumba tango Rumba tango Rumba tango
|
||||
11
edoc/packages-frame.html
Normal file
11
edoc/packages-frame.html
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
||||
<html>
|
||||
<head>
|
||||
<title>The machi application</title>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheet.css" title="EDoc">
|
||||
</head>
|
||||
<body bgcolor="white">
|
||||
<h2 class="indextitle">Packages</h2>
|
||||
<table width="100%" border="0" summary="list of packages"></table>
|
||||
</body>
|
||||
</html>
|
||||
55
edoc/stylesheet.css
Normal file
55
edoc/stylesheet.css
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
/* standard EDoc style sheet */
|
||||
body {
|
||||
font-family: Verdana, Arial, Helvetica, sans-serif;
|
||||
margin-left: .25in;
|
||||
margin-right: .2in;
|
||||
margin-top: 0.2in;
|
||||
margin-bottom: 0.2in;
|
||||
color: #000000;
|
||||
background-color: #ffffff;
|
||||
}
|
||||
h1,h2 {
|
||||
margin-left: -0.2in;
|
||||
}
|
||||
div.navbar {
|
||||
background-color: #add8e6;
|
||||
padding: 0.2em;
|
||||
}
|
||||
h2.indextitle {
|
||||
padding: 0.4em;
|
||||
background-color: #add8e6;
|
||||
}
|
||||
h3.function,h3.typedecl {
|
||||
background-color: #add8e6;
|
||||
padding-left: 1em;
|
||||
}
|
||||
div.spec {
|
||||
margin-left: 2em;
|
||||
background-color: #eeeeee;
|
||||
}
|
||||
a.module,a.package {
|
||||
text-decoration:none
|
||||
}
|
||||
a.module:hover,a.package:hover {
|
||||
background-color: #eeeeee;
|
||||
}
|
||||
ul.definitions {
|
||||
list-style-type: none;
|
||||
}
|
||||
ul.index {
|
||||
list-style-type: none;
|
||||
background-color: #eeeeee;
|
||||
}
|
||||
|
||||
/*
|
||||
* Minor style tweaks
|
||||
*/
|
||||
ul {
|
||||
list-style-type: square;
|
||||
}
|
||||
table {
|
||||
border-collapse: collapse;
|
||||
}
|
||||
td {
|
||||
padding: 3
|
||||
}
|
||||
BIN
images/basho_bench.example0.png
Normal file
BIN
images/basho_bench.example0.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 269 KiB |
BIN
images/supervisor-2flus.png
Normal file
BIN
images/supervisor-2flus.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
|
|
@ -1,63 +0,0 @@
|
|||
%% -------------------------------------------------------------------
|
||||
%%
|
||||
%% Copyright (c) 2007-2015 Basho Technologies, Inc. All Rights Reserved.
|
||||
%%
|
||||
%% This file is provided to you under the Apache License,
|
||||
%% Version 2.0 (the "License"); you may not use this file
|
||||
%% except in compliance with the License. You may obtain
|
||||
%% a copy of the License at
|
||||
%%
|
||||
%% http://www.apache.org/licenses/LICENSE-2.0
|
||||
%%
|
||||
%% Unless required by applicable law or agreed to in writing,
|
||||
%% software distributed under the License is distributed on an
|
||||
%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
%% KIND, either express or implied. See the License for the
|
||||
%% specific language governing permissions and limitations
|
||||
%% under the License.
|
||||
%%
|
||||
%% -------------------------------------------------------------------
|
||||
|
||||
%% @doc Now 4GiBytes, could be up to 64bit due to PB message limit of
|
||||
%% chunk size
|
||||
-define(DEFAULT_MAX_FILE_SIZE, ((1 bsl 32) - 1)).
|
||||
-define(MINIMUM_OFFSET, 1024).
|
||||
|
||||
%% 0th draft of checksum typing with 1st byte.
|
||||
-define(CSUM_TAG_NONE, 0). % No csum provided by client
|
||||
-define(CSUM_TAG_CLIENT_SHA, 1). % Client-generated SHA1
|
||||
-define(CSUM_TAG_SERVER_SHA, 2). % Server-genereated SHA1
|
||||
-define(CSUM_TAG_SERVER_REGEN_SHA, 3). % Server-regenerated SHA1
|
||||
|
||||
-define(CSUM_TAG_NONE_ATOM, none).
|
||||
-define(CSUM_TAG_CLIENT_SHA_ATOM, client_sha).
|
||||
-define(CSUM_TAG_SERVER_SHA_ATOM, server_sha).
|
||||
-define(CSUM_TAG_SERVER_REGEN_SHA_ATOM, server_regen_sha).
|
||||
|
||||
%% Protocol Buffers goop
|
||||
-define(PB_MAX_MSG_SIZE, (33*1024*1024)).
|
||||
-define(PB_PACKET_OPTS, [{packet, 4}, {packet_size, ?PB_MAX_MSG_SIZE}]).
|
||||
|
||||
%% TODO: it's used in flu_sup and elsewhere, change this to suitable name
|
||||
-define(TEST_ETS_TABLE, test_ets_table).
|
||||
|
||||
-define(DEFAULT_COC_NAMESPACE, "").
|
||||
-define(DEFAULT_COC_LOCATOR, 0).
|
||||
|
||||
-record(ns_info, {
|
||||
version = 0 :: machi_dt:namespace_version(),
|
||||
name = <<>> :: machi_dt:namespace(),
|
||||
locator = 0 :: machi_dt:locator()
|
||||
}).
|
||||
|
||||
-record(append_opts, {
|
||||
chunk_extra = 0 :: machi_dt:chunk_size(),
|
||||
preferred_file_name :: 'undefined' | machi_dt:file_name_s(),
|
||||
flag_fail_preferred = false :: boolean()
|
||||
}).
|
||||
|
||||
-record(read_opts, {
|
||||
no_checksum = false :: boolean(),
|
||||
no_chunk = false :: boolean(),
|
||||
needs_trimmed = false :: boolean()
|
||||
}).
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
%% -------------------------------------------------------------------
|
||||
%%
|
||||
%% Copyright (c) 2007-2015 Basho Technologies, Inc. All Rights Reserved.
|
||||
%%
|
||||
%% This file is provided to you under the Apache License,
|
||||
%% Version 2.0 (the "License"); you may not use this file
|
||||
%% except in compliance with the License. You may obtain
|
||||
%% a copy of the License at
|
||||
%%
|
||||
%% http://www.apache.org/licenses/LICENSE-2.0
|
||||
%%
|
||||
%% Unless required by applicable law or agreed to in writing,
|
||||
%% software distributed under the License is distributed on an
|
||||
%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
%% KIND, either express or implied. See the License for the
|
||||
%% specific language governing permissions and limitations
|
||||
%% under the License.
|
||||
%%
|
||||
%% -------------------------------------------------------------------
|
||||
|
||||
-include("machi_projection.hrl").
|
||||
|
||||
-define(NOT_FLAPPING, {0,0,0}).
|
||||
|
||||
-type projection() :: #projection_v1{}.
|
||||
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
%% machi merkle tree records
|
||||
|
||||
-record(naive, {
|
||||
chunk_size = 1048576 :: pos_integer(), %% default 1 MB
|
||||
recalc = true :: boolean(),
|
||||
root :: 'undefined' | binary(),
|
||||
lvl1 = [] :: [ binary() ],
|
||||
lvl2 = [] :: [ binary() ],
|
||||
lvl3 = [] :: [ binary() ],
|
||||
leaves = [] :: [ { Offset :: pos_integer(),
|
||||
Size :: pos_integer(),
|
||||
Csum :: binary()} ]
|
||||
}).
|
||||
|
||||
-record(mt, {
|
||||
filename :: string(),
|
||||
tree :: #naive{},
|
||||
backend = 'naive' :: 'naive'
|
||||
}).
|
||||
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
%% -------------------------------------------------------------------
|
||||
%%
|
||||
%% Copyright (c) 2007-2015 Basho Technologies, Inc. All Rights Reserved.
|
||||
%%
|
||||
%% This file is provided to you under the Apache License,
|
||||
%% Version 2.0 (the "License"); you may not use this file
|
||||
%% except in compliance with the License. You may obtain
|
||||
%% a copy of the License at
|
||||
%%
|
||||
%% http://www.apache.org/licenses/LICENSE-2.0
|
||||
%%
|
||||
%% Unless required by applicable law or agreed to in writing,
|
||||
%% software distributed under the License is distributed on an
|
||||
%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
%% KIND, either express or implied. See the License for the
|
||||
%% specific language governing permissions and limitations
|
||||
%% under the License.
|
||||
%%
|
||||
%% -------------------------------------------------------------------
|
||||
|
||||
-ifndef(MACHI_PROJECTION_HRL).
|
||||
-define(MACHI_PROJECTION_HRL, true).
|
||||
|
||||
-type pv1_consistency_mode() :: 'ap_mode' | 'cp_mode'.
|
||||
-type pv1_chain_name():: atom().
|
||||
-type pv1_csum() :: binary().
|
||||
-type pv1_epoch() :: {pv1_epoch_n(), pv1_csum()}.
|
||||
-type pv1_epoch_n() :: non_neg_integer().
|
||||
-type pv1_server() :: atom().
|
||||
-type pv1_timestamp() :: {non_neg_integer(), non_neg_integer(), non_neg_integer()}.
|
||||
|
||||
-record(p_srvr, {
|
||||
name :: pv1_server(),
|
||||
proto_mod = 'machi_flu1_client' :: atom(), % Module name
|
||||
address :: term(), % Protocol-specific
|
||||
port :: term(), % Protocol-specific
|
||||
props = [] :: list() % proplist for other related info
|
||||
}).
|
||||
|
||||
-record(flap_i, {
|
||||
flap_count :: {term(), term()},
|
||||
all_hosed :: list(),
|
||||
all_flap_counts :: list(),
|
||||
my_unique_prop_count :: non_neg_integer()
|
||||
}).
|
||||
|
||||
-type p_srvr() :: #p_srvr{}.
|
||||
-type p_srvr_dict() :: orddict:orddict().
|
||||
|
||||
-define(DUMMY_PV1_EPOCH, {0,<<0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}).
|
||||
|
||||
%% Kludge for spam gossip. TODO: replace me
|
||||
-define(SPAM_PROJ_EPOCH, ((1 bsl 32) - 7)).
|
||||
|
||||
-record(projection_v1, {
|
||||
epoch_number :: pv1_epoch_n() | ?SPAM_PROJ_EPOCH,
|
||||
epoch_csum :: pv1_csum(),
|
||||
author_server :: pv1_server(),
|
||||
chain_name = ch_not_def_yet :: pv1_chain_name(),
|
||||
all_members :: [pv1_server()],
|
||||
witnesses = [] :: [pv1_server()],
|
||||
creation_time :: pv1_timestamp(),
|
||||
mode = ap_mode :: pv1_consistency_mode(),
|
||||
upi :: [pv1_server()],
|
||||
repairing :: [pv1_server()],
|
||||
down :: [pv1_server()],
|
||||
dbg :: list(), %proplist(), is checksummed
|
||||
dbg2 :: list(), %proplist(), is not checksummed
|
||||
members_dict :: p_srvr_dict()
|
||||
}).
|
||||
|
||||
-define(MACHI_DEFAULT_TCP_PORT, 50000).
|
||||
|
||||
-define(SHA_MAX, (1 bsl (20*8))).
|
||||
|
||||
%% Set a limit to the maximum chain length, so that it's easier to
|
||||
%% create a consistent projection ranking score.
|
||||
-define(MAX_CHAIN_LENGTH, 64).
|
||||
|
||||
-record(chain_def_v1, {
|
||||
name :: atom(), % chain name
|
||||
mode :: pv1_consistency_mode(),
|
||||
full = [] :: [p_srvr()],
|
||||
witnesses = [] :: [p_srvr()],
|
||||
old_full = [] :: [pv1_server()], % guard against some races
|
||||
old_witnesses=[] :: [pv1_server()], % guard against some races
|
||||
local_run = [] :: [pv1_server()], % must be tailored to each machine!
|
||||
local_stop = [] :: [pv1_server()], % must be tailored to each machine!
|
||||
props = [] :: list() % proplist for other related info
|
||||
}).
|
||||
|
||||
-endif. % !MACHI_PROJECTION_HRL
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
%% -------------------------------------------------------------------
|
||||
%%
|
||||
%% Machi: a small village of replicated files
|
||||
%%
|
||||
%% Copyright (c) 2014-2015 Basho Technologies, Inc. All Rights Reserved.
|
||||
%%
|
||||
%% This file is provided to you under the Apache License,
|
||||
%% Version 2.0 (the "License"); you may not use this file
|
||||
%% except in compliance with the License. You may obtain
|
||||
%% a copy of the License at
|
||||
%%
|
||||
%% http://www.apache.org/licenses/LICENSE-2.0
|
||||
%%
|
||||
%% Unless required by applicable law or agreed to in writing,
|
||||
%% software distributed under the License is distributed on an
|
||||
%% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
%% KIND, either express or implied. See the License for the
|
||||
%% specific language governing permissions and limitations
|
||||
%% under the License.
|
||||
%%
|
||||
%% -------------------------------------------------------------------
|
||||
|
||||
-ifdef(PULSE).
|
||||
-define(V(Fmt, Args), pulse:format(Fmt, Args)).
|
||||
-else. % PULSE
|
||||
-define(V(Fmt, Args), io:format(user, Fmt, Args)).
|
||||
-endif. % PULSE
|
||||
|
||||
-define(D(X), ?V("~s ~p\n", [??X, X])).
|
||||
-define(Dw(X), ?V("~s ~w\n", [??X, X])).
|
||||
|
||||
38
index.html
Normal file
38
index.html
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en-us">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>Machi by basho</title>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<link rel="stylesheet" type="text/css" href="stylesheets/normalize.css" media="screen">
|
||||
<link href='http://fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
|
||||
<link rel="stylesheet" type="text/css" href="stylesheets/stylesheet.css" media="screen">
|
||||
<link rel="stylesheet" type="text/css" href="stylesheets/github-light.css" media="screen">
|
||||
</head>
|
||||
<body>
|
||||
<section class="page-header">
|
||||
<h1 class="project-name">Machi</h1>
|
||||
<h2 class="project-tagline">Machi file store: developer API</h2>
|
||||
</section>
|
||||
|
||||
<section class="main-content">
|
||||
<p> <a href="edoc/">For a snapshot view of the HTML "EDoc" documentation for Machi, please follow this EDoc Link</a>.</p>
|
||||
<p> <a href="https://github.com/basho/machi/tree/master/doc/">All other Machi documentation.</a></p>
|
||||
<p> <a href="https://github.com/basho/machi">View README & source on GitHub</a>
|
||||
<p> <a href="https://github.com/basho/machi/zipball/master">Download
|
||||
source as a ZIP file</a>
|
||||
<p> <a href="https://github.com/basho/machi/tarball/master">Download
|
||||
source as a compressed TAR file</a>
|
||||
|
||||
<footer class="site-footer">
|
||||
<span class="site-footer-owner"><a href="https://github.com/basho/machi">Machi</a> is maintained by <a href="https://github.com/basho">basho</a>.</span>
|
||||
|
||||
<span class="site-footer-credits">This page was generated by <a href="https://pages.github.com">GitHub Pages</a> using the <a href="https://github.com/jasonlong/cayman-theme">Cayman theme</a> by <a href="https://twitter.com/jasonlong">Jason Long</a>.</span>
|
||||
</footer>
|
||||
|
||||
</section>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
1
params.json
Normal file
1
params.json
Normal file
|
|
@ -0,0 +1 @@
|
|||
{"name":"Machi","tagline":"Machi file store: developer API","body":"For a snapshot view of the HTML \"EDoc\" documentation for Machi, please follow the [EDoc Link](edoc/).","google":"","note":"Don't delete this file! It's used internally to help with page regeneration."}
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
%% Mandatory: adjust this code path to top of your compiled Machi source distro
|
||||
{code_paths, ["/Users/fritchie/b/src/machi"]}.
|
||||
{driver, machi_basho_bench_driver}.
|
||||
|
||||
%% Chose your maximum rate (per worker proc, see 'concurrent' below)
|
||||
%{mode, {rate,10}}.
|
||||
%{mode, {rate,20}}.
|
||||
{mode, max}.
|
||||
|
||||
%% Runtime & reporting interval
|
||||
{duration, 10}. % minutes
|
||||
{report_interval, 1}. % seconds
|
||||
|
||||
%% Choose your number of worker procs
|
||||
%{concurrent, 1}.
|
||||
{concurrent, 5}.
|
||||
%{concurrent, 10}.
|
||||
|
||||
%% Here's a chain of (up to) length 3, all on localhost
|
||||
%% Note: if any servers are down, and your OS/TCP stack has an
|
||||
%% ICMP response limit such as OS X's "net.inet.icmp.icmplim" setting,
|
||||
%% then if that setting is very low (e.g., OS X's limit is 50), then
|
||||
%% you can have big problems with ICMP/RST responses being delayed and
|
||||
%% interactive *very* badly with your test.
|
||||
%% For OS X, fix using "sudo sysctl -w net.inet.icmp.icmplim=9999"
|
||||
{machi_server_info,
|
||||
[
|
||||
{p_srvr,a,machi_flu1_client,"localhost",4444,[]},
|
||||
{p_srvr,b,machi_flu1_client,"localhost",4445,[]},
|
||||
{p_srvr,c,machi_flu1_client,"localhost",4446,[]}
|
||||
]}.
|
||||
{machi_ets_key_tab_type, set}. % set | ordered_set
|
||||
|
||||
%% Workload-specific definitions follow....
|
||||
|
||||
%% 10 parts 'append' operation + 0 parts anything else = 100% 'append' ops
|
||||
{operations, [{append, 10}]}.
|
||||
|
||||
%% For append, key = Machi file prefix name
|
||||
{key_generator, {to_binstr, "prefix~w", {uniform_int, 30}}}.
|
||||
|
||||
%% Increase size of value_generator_source_size if value_generator is big!!
|
||||
{value_generator_source_size, 2111000}.
|
||||
{value_generator, {fixed_bin, 32768}}. % 32 KB
|
||||
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
%% Mandatory: adjust this code path to top of your compiled Machi source distro
|
||||
{code_paths, ["/Users/fritchie/b/src/machi"]}.
|
||||
{driver, machi_basho_bench_driver}.
|
||||
|
||||
%% Chose your maximum rate (per worker proc, see 'concurrent' below)
|
||||
%{mode, {rate,10}}.
|
||||
%{mode, {rate,20}}.
|
||||
{mode, max}.
|
||||
|
||||
%% Runtime & reporting interval
|
||||
{duration, 10}. % minutes
|
||||
{report_interval, 1}. % seconds
|
||||
|
||||
%% Choose your number of worker procs
|
||||
%{concurrent, 1}.
|
||||
{concurrent, 5}.
|
||||
%{concurrent, 10}.
|
||||
|
||||
%% Here's a chain of (up to) length 3, all on localhost
|
||||
%% Note: if any servers are down, and your OS/TCP stack has an
|
||||
%% ICMP response limit such as OS X's "net.inet.icmp.icmplim" setting,
|
||||
%% then if that setting is very low (e.g., OS X's limit is 50), then
|
||||
%% you can have big problems with ICMP/RST responses being delayed and
|
||||
%% interactive *very* badly with your test.
|
||||
%% For OS X, fix using "sudo sysctl -w net.inet.icmp.icmplim=9999"
|
||||
{machi_server_info,
|
||||
[
|
||||
{p_srvr,a,machi_flu1_client,"localhost",4444,[]},
|
||||
{p_srvr,b,machi_flu1_client,"localhost",4445,[]},
|
||||
{p_srvr,c,machi_flu1_client,"localhost",4446,[]}
|
||||
]}.
|
||||
{machi_ets_key_tab_type, set}. % set | ordered_set
|
||||
|
||||
%% Workload-specific definitions follow....
|
||||
|
||||
%% 10 parts 'read' operation + 0 parts anything else = 100% 'read' ops
|
||||
{operations, [{read, 10}]}.
|
||||
|
||||
%% For read, key = integer index into Machi's chunk ETS table, modulo the
|
||||
%% ETS table size, so a huge number here is OK.
|
||||
{key_generator, {uniform_int, 999999999999}}.
|
||||
|
||||
%% For read, value_generator_* isn't used, so leave these defaults as-is.
|
||||
{value_generator_source_size, 2111000}.
|
||||
{value_generator, {fixed_bin, 32768}}. % 32 KB
|
||||
|
||||
|
||||
|
|
@ -1,56 +0,0 @@
|
|||
#!/bin/sh
|
||||
|
||||
echo "Step: Verify that the required entries in /etc/hosts are present"
|
||||
for i in 1 2 3; do
|
||||
grep machi$i /etc/hosts | egrep -s '^127.0.0.1' > /dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo ""
|
||||
echo "'grep -s machi$i' failed. Aborting, sorry."
|
||||
exit 1
|
||||
fi
|
||||
ping -c 1 machi$i > /dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo ""
|
||||
echo "Ping attempt on host machi$i failed. Aborting."
|
||||
echo ""
|
||||
ping -c 1 machi$i
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Step: add a verbose logging option to app.config"
|
||||
for i in 1 2 3; do
|
||||
ed ./dev/dev$i/etc/app.config <<EOF > /dev/null 2>&1
|
||||
/verbose_confirm
|
||||
a
|
||||
{chain_manager_opts, [{private_write_verbose_confirm,true}]},
|
||||
{stability_time, 1},
|
||||
.
|
||||
w
|
||||
q
|
||||
EOF
|
||||
done
|
||||
|
||||
echo "Step: start three three Machi application instances"
|
||||
for i in 1 2 3; do
|
||||
./dev/dev$i/bin/machi start
|
||||
./dev/dev$i/bin/machi ping
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Sorry, a 'ping' check for instance dev$i failed. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Step: configure one chain to start a Humming Consensus group with three members"
|
||||
|
||||
# Note: $CWD of each Machi proc is two levels below the source code root dir.
|
||||
LIFECYCLE000=../../priv/quick-admin-examples/demo-000
|
||||
for i in 3 2 1; do
|
||||
./dev/dev$i/bin/machi-admin quick-admin-apply $LIFECYCLE000 machi$i
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Sorry, 'machi-admin quick-admin-apply failed' on machi$i. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
exit 0
|
||||
93
priv/humming-consensus-demo.vagrant/Vagrantfile
vendored
93
priv/humming-consensus-demo.vagrant/Vagrantfile
vendored
|
|
@ -1,93 +0,0 @@
|
|||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
|
||||
# All Vagrant configuration is done below. The "2" in Vagrant.configure
|
||||
# configures the configuration version (we support older styles for
|
||||
# backwards compatibility). Please don't change it unless you know what
|
||||
# you're doing.
|
||||
Vagrant.configure(2) do |config|
|
||||
# The most common configuration options are documented and commented below.
|
||||
# For a complete reference, please see the online documentation at
|
||||
# https://docs.vagrantup.com.
|
||||
|
||||
# Every Vagrant development environment requires a box. You can search for
|
||||
# boxes at https://atlas.hashicorp.com/search.
|
||||
# If this Vagrant box has not been downloaded before (e.g. using "vagrant box add"),
|
||||
# then Vagrant will automatically download the VM image from HashiCorp.
|
||||
config.vm.box = "hashicorp/precise64"
|
||||
# If using a FreeBSD box, Bash may not be installed.
|
||||
# Use the config.ssh.shell setting to specify an alternate shell.
|
||||
# Note, however, that any code in the 'config.vm.provision' section
|
||||
# would then have to use this shell's syntax!
|
||||
# config.ssh.shell = "/bin/csh -l"
|
||||
|
||||
# Disable automatic box update checking. If you disable this, then
|
||||
# boxes will only be checked for updates when the user runs
|
||||
# `vagrant box outdated`. This is not recommended.
|
||||
# config.vm.box_check_update = false
|
||||
|
||||
# Create a forwarded port mapping which allows access to a specific port
|
||||
# within the machine from a port on the host machine. In the example below,
|
||||
# accessing "localhost:8080" will access port 80 on the guest machine.
|
||||
# config.vm.network "forwarded_port", guest: 80, host: 8080
|
||||
|
||||
# Create a private network, which allows host-only access to the machine
|
||||
# using a specific IP.
|
||||
# config.vm.network "private_network", ip: "192.168.33.10"
|
||||
|
||||
# Create a public network, which generally matched to bridged network.
|
||||
# Bridged networks make the machine appear as another physical device on
|
||||
# your network.
|
||||
# config.vm.network "public_network"
|
||||
|
||||
# Share an additional folder to the guest VM. The first argument is
|
||||
# the path on the host to the actual folder. The second argument is
|
||||
# the path on the guest to mount the folder. And the optional third
|
||||
# argument is a set of non-required options.
|
||||
# config.vm.synced_folder "../data", "/vagrant_data"
|
||||
|
||||
# Provider-specific configuration so you can fine-tune various
|
||||
# backing providers for Vagrant. These expose provider-specific options.
|
||||
# Example for VirtualBox:
|
||||
#
|
||||
config.vm.provider "virtualbox" do |vb|
|
||||
# Display the VirtualBox GUI when booting the machine
|
||||
# vb.gui = true
|
||||
|
||||
# Customize the amount of memory on the VM:
|
||||
vb.memory = "512"
|
||||
end
|
||||
#
|
||||
# View the documentation for the provider you are using for more
|
||||
# information on available options.
|
||||
|
||||
# Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
|
||||
# such as FTP and Heroku are also available. See the documentation at
|
||||
# https://docs.vagrantup.com/v2/push/atlas.html for more information.
|
||||
# config.push.define "atlas" do |push|
|
||||
# push.app = "YOUR_ATLAS_USERNAME/YOUR_APPLICATION_NAME"
|
||||
# end
|
||||
|
||||
# Enable provisioning with a shell script. Additional provisioners such as
|
||||
# Puppet, Chef, Ansible, Salt, and Docker are also available. Please see the
|
||||
# documentation for more information about their specific syntax and use.
|
||||
config.vm.provision "shell", inline: <<-SHELL
|
||||
# Install prerequsites
|
||||
# Support here for FreeBSD is experimental
|
||||
apt-get update ; sudo apt-get install -y git sudo rsync ; # Ubuntu Linux
|
||||
env ASSUME_ALWAYS_YES=yes pkg install -f git sudo rsync ; # FreeBSD 10
|
||||
|
||||
# Install dependent packages, using slf-configurator
|
||||
git clone https://github.com/slfritchie/slf-configurator.git
|
||||
chown -R vagrant ./slf-configurator
|
||||
(cd slf-configurator ; sudo sh -x ./ALL.sh)
|
||||
echo 'export PATH=${PATH}:/usr/local/erlang/17.5/bin' >> ~vagrant/.bashrc
|
||||
export PATH=${PATH}:/usr/local/erlang/17.5/bin
|
||||
## echo 'set path = ( $path /usr/local/erlang/17.5/bin )' >> ~vagrant/.cshrc
|
||||
## setenv PATH /usr/local/erlang/17.5/bin:$PATH
|
||||
|
||||
git clone https://github.com/basho/machi.git
|
||||
(cd machi ; git checkout master ; make && make test )
|
||||
chown -R vagrant ./machi
|
||||
SHELL
|
||||
end
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
#!/usr/bin/perl
|
||||
|
||||
$input = shift;
|
||||
$tmp1 = "/tmp/my-tmp.1.$$";
|
||||
$tmp2 = "/tmp/my-tmp.2.$$";
|
||||
$l1 = 0;
|
||||
$l2 = 0;
|
||||
$l3 = 0;
|
||||
|
||||
open(I, $input);
|
||||
open(T1, "> $tmp1");
|
||||
open(T2, "> $tmp2");
|
||||
|
||||
while (<I>) {
|
||||
if (/^##*/) {
|
||||
$line = $_;
|
||||
chomp;
|
||||
@a = split;
|
||||
$count = length($a[0]) - 2;
|
||||
if ($count >= 0) {
|
||||
if ($count == 0) {
|
||||
$l1++;
|
||||
$l2 = 0;
|
||||
$l3 = 0;
|
||||
$label = "$l1"
|
||||
}
|
||||
if ($count == 1) {
|
||||
$l2++;
|
||||
$l3 = 0;
|
||||
$label = "$l1.$l2"
|
||||
}
|
||||
if ($count == 2) {
|
||||
$l3++;
|
||||
$label = "$l1.$l2.$l3"
|
||||
}
|
||||
$indent = " " x ($count * 4);
|
||||
s/^#*\s*[0-9. ]*//;
|
||||
$anchor = "n$label";
|
||||
printf T1 "%s+ [%s. %s](#%s)\n", $indent, $label, $_, $anchor;
|
||||
printf T2 "<a name=\"%s\">\n", $anchor;
|
||||
$line =~ s/(#+)\s*[0-9. ]*/$1 $label. /;
|
||||
print T2 $line;
|
||||
} else {
|
||||
print T2 $_, "\n";
|
||||
}
|
||||
} else {
|
||||
next if /^<a name="n[0-9.]+">/;
|
||||
print T2 $_;
|
||||
}
|
||||
}
|
||||
|
||||
close(I);
|
||||
close(T1);
|
||||
close(T2);
|
||||
open(T2, $tmp2);
|
||||
|
||||
while (<T2>) {
|
||||
if (/<!\-\- OUTLINE \-\->/) {
|
||||
print;
|
||||
print "\n";
|
||||
open(T1, $tmp1);
|
||||
while (<T1>) {
|
||||
print;
|
||||
}
|
||||
close(T1);
|
||||
while (<T2>) {
|
||||
if (/<!\-\- ENDOUTLINE \-\->/) {
|
||||
print "\n";
|
||||
print;
|
||||
last;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
print;
|
||||
}
|
||||
}
|
||||
close(T2);
|
||||
|
||||
unlink($tmp1);
|
||||
unlink($tmp2);
|
||||
exit(0);
|
||||
|
|
@ -1 +0,0 @@
|
|||
{host, "localhost", []}.
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
{flu,f1,"localhost",20401,[]}.
|
||||
{flu,f2,"localhost",20402,[]}.
|
||||
{flu,f3,"localhost",20403,[]}.
|
||||
{chain,c1,[f1,f2,f3],[]}.
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
{flu,f4,"localhost",20404,[]}.
|
||||
{flu,f5,"localhost",20405,[]}.
|
||||
{flu,f6,"localhost",20406,[]}.
|
||||
{chain,c2,[f4,f5,f6],[]}.
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
{host, "machi1", []}.
|
||||
{host, "machi2", []}.
|
||||
{host, "machi3", []}.
|
||||
{flu,f1,"machi1",20401,[]}.
|
||||
{flu,f2,"machi2",20402,[]}.
|
||||
{flu,f3,"machi3",20403,[]}.
|
||||
{chain,c1,[f1,f2,f3],[]}.
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue