greg/machi
1
0
Fork 0
Code Issues 11 Pull requests 1 Projects Releases Packages Wiki Activity Actions
546 commits 58 branches 0 tags 5.9 MiB
Commit graph

185 commits

Author SHA1 Message Date
Scott Lystig Fritchie
d3b0b7fdc5 Clean up some dialyzer complaints 2015-06-23 17:26:15 +09:00
Scott Lystig Fritchie
727b2a987d ROTFL forgot to add src/machi_pb_server.erl 2015-06-23 17:22:45 +09:00
Scott Lystig Fritchie
ceebe3d491 WIP: list_files #2 2015-06-23 17:17:14 +09:00
Scott Lystig Fritchie
73f71c406e WIP: list_files end-to-end! 2015-06-23 17:08:15 +09:00
Scott Lystig Fritchie
6722b3c0f1 WIP: checksum_list incomplete implementation.... 2015-06-23 16:53:06 +09:00
Scott Lystig Fritchie
44c22bf752 WIP: read_chunk #1 2015-06-23 15:34:48 +09:00
Scott Lystig Fritchie
a8782eed5a WIP: write_chunk #1 2015-06-23 15:13:13 +09:00
Scott Lystig Fritchie
cb06c53dc0 WIP: PB append_chunk end-to-end works! 2015-06-23 14:45:24 +09:00
Scott Lystig Fritchie
5ef499ec73 WIP: append_chunk #1 2015-06-23 14:08:10 +09:00
Scott Lystig Fritchie
bb8e725c26 WIP: 'auth' request placeholders 2015-06-22 18:16:15 +09:00
Scott Lystig Fritchie
db7f1476b9 WIP: 'echo' request works end-to-end, yay! 2015-06-22 18:04:17 +09:00
Scott Lystig Fritchie
3d05f543df WIP: new test case is failing, quick fix soon 2015-06-22 17:49:07 +09:00
Scott Lystig Fritchie
dc9f272c44 Nearly dumbest-possible Protocol Buffers client request & response round trip 2015-06-19 17:21:04 +09:00
Scott Lystig Fritchie
c4bdeee4da Oops, add missing src/machi_dt.erl 2015-06-19 17:20:49 +09:00
Scott Lystig Fritchie
0cdaee32f8 Egadz, edoc doesn't use preprocessor {sigh} 2015-06-19 16:24:57 +09:00
Scott Lystig Fritchie
984b4f7a86 Dialyzer tightening and subsequent cleanup 2015-06-19 16:04:34 +09:00
Scott Lystig Fritchie
1372bd9594 {sigh} add filter-dialyzer-dep-warnings 2015-06-19 15:22:07 +09:00
Scott Lystig Fritchie
3c300bb9f1 Add write_chunk() to machi_cr_client.erl 2015-06-19 14:49:09 +09:00
Scott Lystig Fritchie
40c0a72b48 Add test/machi_pb_test.erl, finish PB refactoring 2015-06-19 13:00:28 +09:00
Scott Lystig Fritchie
a82bd68f3c Overhaul the 0.1 PB definition. Again. 
Many thanks to @seancribbs for a suggestion to avoid the PB design
mistake/feature of the original Riak KV PB API.
 2015-06-19 12:28:31 +09:00
Scott Lystig Fritchie
87b636a349 WIP: PB wrestling 2015-06-18 17:31:48 +09:00
Scott Lystig Fritchie
37a8c1c124 WIP: PB wrestling 2015-06-18 16:16:23 +09:00
Scott Lystig Fritchie
e5673b5e20 First attempt at Protocol Buffers .proto + infrastructure 2015-06-17 16:12:20 +09:00
Scott Lystig Fritchie
22337e1819 Remove short circuit (bad idea!) from react_to_env_C100() 2015-06-15 17:22:02 +09:00
Scott Lystig Fritchie
b244a3b8e4 Reduce verbosity, try fix up convergence demo for chain len=4 2015-06-15 12:41:16 +09:00
Scott Lystig Fritchie
9bf76e0bfb Fix for correctness bug, thanks PULSE 2015-06-05 01:06:39 +09:00
Scott Lystig Fritchie
be62300b3b Bug fixes: model and real bugs, thanks PULSE and converge_demo both! 2015-06-04 17:39:29 +09:00
Scott Lystig Fritchie
0cf9627f26 Bugfix, found by inspection, yay! 2015-06-04 15:05:37 +09:00
Scott Lystig Fritchie
89b8b6a012 Bugfix, found by PULSE, yay! 2015-06-04 14:31:58 +09:00
Scott Lystig Fritchie
d3df2bd31d WIP: remove repair_always_done option, it was flawed 2015-06-03 15:26:22 +09:00
Scott Lystig Fritchie
87417d2872 WIP: get the old jalopy into runnable shape 2015-06-03 11:48:55 +09:00
Scott Lystig Fritchie
2207151eba Fix projection_transition_is_sane() bug 2015-06-02 21:20:50 +09:00
Scott Lystig Fritchie
deabe14d29 Un-proplist-ify the inner projection 2015-06-02 20:55:18 +09:00
Scott Lystig Fritchie
207be8729b Un-proplist-ify the flapping_i info 2015-06-02 20:32:52 +09:00
Scott Lystig Fritchie
0f10b45161 Dialyzer fixes, derp! 2015-06-02 19:07:13 +09:00
Scott Lystig Fritchie
000d687588 Fix creation_time bug in inner projection 2015-06-02 16:26:49 +09:00
Scott Lystig Fritchie
cffbd3c50c Add checksum handling strawman to strawman HTTP interface 2015-06-02 13:23:36 +09:00
Scott Lystig Fritchie
dd4160b963 Add basic {error, bad_checksum} tests to proxy & CR clients 2015-06-02 12:36:51 +09:00
Scott Lystig Fritchie
e3162fdcda Rudimentary client-side checksum and server-side checksum type tags 2015-06-01 14:25:55 +09:00
Scott Lystig Fritchie
6cebf39723 Damn ugly HTTP interface "equivalent" for machi_cr_client.erl basic API 
This goes to show that mixing implementation and protocol and API
and lots of other stuff ... is cool for the quick hack to do one thing
but really sucks when trying to do more than one thing.

* Proof-of-concept only: add HTTP/1.0'ish 'PUT' interface to be the
rough equivalent of machi_cr_client:append_chunk/3
* Proof-of-concept only: add HTTP/1.0'ish 'GET' interface to be the
rough equivalent of machi_cr_client:read_chunk/4

Example use: `append_chunk`

    % curl http://127.0.0.1:4444/foo -0 -T /etc/hosts -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > PUT /foo HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    > Content-Length: 338
    >
    * We are completely uploaded and fine
    * HTTP 1.0, assume close after body
    < HTTP/1.0 201 Created
    < Location: foo.50EI18AX.21
    < X-Offset: 3052
    < X-Size: 338
    <
    * Closing connection 0

Example_use: `read_chunk`

    curl 'http://127.0.0.1:4444/foo.50EI18AX.21?offset=3052&size=338' -0 -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > GET /foo.50EI18AX.21?offset=3052&size=338 HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    >
    * HTTP 1.0, assume close after body
    < HTTP/1.0 200 OK
    < Content-Length: 338
    <
    ##
    # Host Database
    #
    # localhost is used to configure the loopback interface
    # when the system is booting.  Do not change this entry.
    ##
    127.0.0.1	localhost
    127.0.0.1	test.localhost
    255.255.255.255	broadcasthost
    ::1             localhost
    fe80::1%lo0	localhost

    # Xxxxxxx Yyyyy
    192.168.99.222	zzzzz

    127.0.0.1   aaaaaaaa.bb.ccccccccc.com
    * Closing connection 0
 2015-05-22 17:51:06 +09:00
Scott Lystig Fritchie
7ba962e9fb Fix handling of {error, partial_read} 2015-05-21 15:12:46 +09:00
Scott Lystig Fritchie
9e41162e65 Minor machi_basho_bench_driver changes 2015-05-20 18:44:04 +09:00
Scott Lystig Fritchie
1c13273efd Add simple basho_bench driver, no schmancy bells or whistles 2015-05-20 17:30:37 +09:00
Scott Lystig Fritchie
69244691f4 Such wonder when one *reads* the docs... 2015-05-20 14:12:48 +09:00
Scott Lystig Fritchie
b5ddfaf019 Finish basic API for machi_cr_client.erl 2015-05-19 20:04:36 +09:00
Scott Lystig Fritchie
a4266e8aa4 Fix known chain repair bugs, add basic smoke test 2015-05-19 19:32:48 +09:00
Scott Lystig Fritchie
152e487060 WIP: read-repair, new test is failing, yay 2015-05-19 15:15:05 +09:00
Scott Lystig Fritchie
079d15dd5c Derp, remove debugging goop + fix eunit @ write_chunk() response change 2015-05-19 14:05:18 +09:00
Scott Lystig Fritchie
eaf007ec08 Fix read repair FLU tracking 2015-05-19 13:56:12 +09:00
Scott Lystig Fritchie
f7274e7106 WIP: brute-force read-repair 2015-05-18 23:26:21 +09:00
Scott Lystig Fritchie
185c670b2f WIP: refactoring machi_cr_client:append_chunk* 2015-05-18 19:06:06 +09:00
Scott Lystig Fritchie
a347722a15 Fix {error,not_written} type bugs in chmgr 2015-05-18 17:32:22 +09:00
Scott Lystig Fritchie
966d2edae8 WIP: refactoring machi_cr_client:append_chunk* 2015-05-18 15:49:05 +09:00
Scott Lystig Fritchie
a7f53cf21a WIP: starting machi_cr_client:append_chunk* 2015-05-18 00:59:24 +09:00
Scott Lystig Fritchie
b0607ae815 WIP: starting machi_cr_client:append_chunk* 2015-05-18 00:33:15 +09:00
Scott Lystig Fritchie
d293170e92 WIP: starting machi_cr_client.erl 2015-05-17 23:48:05 +09:00
Scott Lystig Fritchie
10364834de Add a dummy client-side implementation module:machi_yessir_client.erl 2015-05-17 19:00:51 +09:00
Scott Lystig Fritchie
d4080b78d8 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:46:50 +09:00
Scott Lystig Fritchie
a8c5879d21 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:25:58 +09:00
Scott Lystig Fritchie
a06055ac23 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:18:30 +09:00
Scott Lystig Fritchie
c7d4131a44 Remove unused verb() 2015-05-17 14:19:37 +09:00
Scott Lystig Fritchie
6c07522359 Add new API func, append_chunk_extra() 2015-05-17 14:10:42 +09:00
Scott Lystig Fritchie
5c2635346f Basic multi-party chain repair for ap_mode finished 2015-05-16 17:39:58 +09:00
Scott Lystig Fritchie
d2f1549aa3 WIP: more generic all-way file chunk merge func, part 3 2015-05-16 17:11:54 +09:00
Scott Lystig Fritchie
04bc28b9da WIP: more generic all-way file chunk merge func, part 2 2015-05-16 16:55:48 +09:00
Scott Lystig Fritchie
a9c753ad64 WIP: more generic all-way file chunk merge func 2015-05-15 17:15:02 +09:00
Scott Lystig Fritchie
358764d403 WIP: repair common file, part 0 2015-05-14 14:04:31 +09:00
Scott Lystig Fritchie
19d3c95325 WIP: aside, damn, add missing hex encoding for epochid, derp 2015-05-13 18:57:38 +09:00
Scott Lystig Fritchie
eec029b08f WIP: aside, fix FLU wedge status @ init() 2015-05-13 17:59:32 +09:00
Scott Lystig Fritchie
4ae0f94649 WIP: move to stats via ETS, success/failure propagates, yay! 2015-05-12 23:45:35 +09:00
Scott Lystig Fritchie
cad84442bb WIP: stats record, hrm 2015-05-12 22:42:03 +09:00
Scott Lystig Fritchie
8807f954ff WIP: Whole file repair is 95% complete, yay! 2015-05-12 21:45:40 +09:00
Scott Lystig Fritchie
f48720e4dc WIP: set up proxies for repair 2015-05-12 12:56:41 +09:00
Scott Lystig Fritchie
1c70a46b09 Add basic process & bookkeeping structure for repair proc 
=INFO REPORT==== 11-May-2015::19:50:09 ===
    Chain tail a of [a] starting repair of [c]

    =INFO REPORT==== 11-May-2015::19:50:12 ===
    Chain tail a of [a]: repair finished in 2.438 seconds: todo_yo
 2015-05-11 19:50:21 +09:00
Scott Lystig Fritchie
c82000dc30 Reduce spamminess slightly 2015-05-11 19:00:21 +09:00
Scott Lystig Fritchie
33bfbe109e Chain manager bug fixes & enhancment (more...) 
* Set max length of a chain at -define(MAX_CHAIN_LENGTH, 64).

* Perturb tick sleep time of each manager

* If a chain manager L has zero members in its chain, and then its local
public projection store (authored by some remote author R) has a projection
that contains L, then adopt R's projection and start humming consensus.

* Handle "cross-talk" across projection stores, when chain membership
is changed administratively, e.g. chain was [a,b,c] then changed to merely
[a], but that change only happens on a.  Servers b & c continue to use
stale projections and scribble their projection suggestions to a, causing
it to flap.

What's really cool about the flapping handling is that it *works*.  I
wasn't thinking about this scenario when designing the flapping logic, but
it's really nifty that this extra scenario causes a to flap and then a's
inner projection remains stable, yay!

* Add complaints when "cross-talk" is observed.

* Fix flapping sleep time throttle.

* Fix bug in the machi_projection_store.erl's bookkeeping of the
max epoch number when flapping.
 2015-05-11 18:41:45 +09:00
Scott Lystig Fritchie
dc43a32945 WIP: tests for wedge state all working 2015-05-08 21:37:19 +09:00
Scott Lystig Fritchie
0dd9282789 WIP: fix other broken eunit tests, surrounding wedge state 2015-05-08 21:24:07 +09:00
Scott Lystig Fritchie
6f7818fca7 WIP: additional tests for wedge state 2015-05-08 19:50:47 +09:00
Scott Lystig Fritchie
316126fa59 WIP: additional tests for wedge state 2015-05-08 19:07:57 +09:00
Scott Lystig Fritchie
7906e6c235 WIP: basic wedge notifications now working 2015-05-08 18:17:41 +09:00
Scott Lystig Fritchie
d6d003618d WIP: add wedge_status() query to proxy client 2015-05-08 16:58:06 +09:00
Scott Lystig Fritchie
1dc759b908 WIP: add wedge_status() query to client 2015-05-08 16:53:10 +09:00
Scott Lystig Fritchie
762aef557f WIP: Set the stage for FLU wedging API 2015-05-08 15:36:53 +09:00
Scott Lystig Fritchie
ae1d038abe Change default value of chmgr's use_partition_simulator to false 2015-05-08 13:40:44 +09:00
Scott Lystig Fritchie
238c8472cd WIP: timeout comments 2015-05-07 18:52:01 +09:00
Scott Lystig Fritchie
14fc37bd0d Add ability to start FLUs at application startup 2015-05-07 18:39:39 +09:00
Scott Lystig Fritchie
517941aaaa Finish chain manager restart & membership changing 2015-05-07 17:52:16 +09:00
Scott Lystig Fritchie
aeb2e4ef9e WIP: partial refactoring of chmgr 2nd start code, one test broken 2015-05-06 11:41:04 +09:00
Scott Lystig Fritchie
a7bd8e43d3 Clean up machi_flu_psup_test.erl 2015-05-02 17:10:23 +09:00
Scott Lystig Fritchie
1675020150 WIP, tests pass again, includign the newest one 2015-05-02 00:33:49 +09:00
Scott Lystig Fritchie
53f6a753f4 WIP: tests pass, but not finished yet 2015-05-01 14:51:42 +09:00
Scott Lystig Fritchie
7bafc1c28a WIP: stop for the night, we are broken 2015-04-30 23:16:08 +09:00
Scott Lystig Fritchie
99fd7e7fe1 Add test/machi_flu_psup_test.erl, small cleanups 2015-04-30 21:20:21 +09:00
Scott Lystig Fritchie
442e79e4f1 Add machi_flu_psup.erl to supervise all 3 FLU processes (see below) 
Introduce machi_flu_psup:start_flu_package/4 as a way to start all
related FLU processes
    * The projection store
    * The chain manager
    * The FLU itself

... as well as linked processes.
http://www.snookles.com/scotttmp/flu-tree-20150430.png shows one FLU
running, "a".  The process registered "a" is the append server,
"some-prefix" for the sequencer & writer for the current <<"some-prefix">>
file, and a process each for 3 active TCP connections to that FLU.
 2015-04-30 19:15:27 +09:00
Scott Lystig Fritchie
9249663ff6 TODO updates for machi_flu1.erl based on doc rewriting work 2015-04-30 15:15:43 +09:00
Scott Lystig Fritchie
02bc7fe0bc WIP: Fix bug that flaps inside an inner projection, oops! 2015-04-14 18:23:00 +09:00
Scott Lystig Fritchie
90df655256 WIP: Ha! There's a bug, this verbose logging change made it easier to see 2015-04-14 16:38:19 +09:00
Scott Lystig Fritchie
9e587b3d11 WIP: crufty TODO & comment cleanup 2015-04-14 16:17:49 +09:00
Scott Lystig Fritchie
59936eda62 WIP: By Jove, I believe the chain manager is working 2015-04-14 15:30:24 +09:00