greg/machi
1
0
Fork 0
Code Issues 11 Pull requests 1 Projects Releases Packages Wiki Activity Actions
1117 commits 58 branches 0 tags 5.9 MiB
Commit graph

322 commits

Author SHA1 Message Date
Scott Lystig Fritchie
9b0a5a1dc3 WIP: 1st part of moving old chain state transtion code to new 
Ha, famous last words, amirite?

    %% The chain sequence/order checks at the bottom of this function aren't
    %% as easy-to-read as they ought to be.  However, I'm moderately confident
    %% that it isn't buggy.  TODO: refactor them for clarity.

So, now machi_chain_manager1:projection_transition_is_sane() is using
newer, far less buggy code to make sanity decisions.

TODO: Add support for Retrospective mode. TODO is it really needed?

Examples of how the old code sucks and the new code sucks less.

    138> eqc:quickcheck(eqc:testing_time(10, machi_chain_manager1_test:prop_compare_legacy_with_v2_chain_transition_check(whole))).
    xxxxxxxxxxxx..x.xxxxxx..x.x....x..xx........................................................Failed! After 69 tests.
    [a,b,c]
    {c,[a,b,c],[c,b],b,[b,a],[b,a,c]}
    Old_res ([335,192,166,160,153,139]): true
    New_res: false (why line [1936])
    Shrinking xxxxxxxxxxxx.xxxxxxx.xxx.xxxxxxxxxxxxxxxxx(3 times)
    [a,b,c]
 %% {Author1,UPI1,   Repair1,Author2,UPI2, Repair2} %%
    {c,      [a,b,c],[],     a,      [b,a],[]}
    Old_res ([338,185,160,153,147]): true
    New_res: false (why line [1936])
    false

Old code is wrong: we've swapped order of a & b, which is bad.

    139> eqc:quickcheck(eqc:testing_time(10, machi_chain_manager1_test:prop_compare_legacy_with_v2_chain_transition_check(whole))).
    xxxxxxxxxx..x...xx..........xxx..x..............x......x............................................(x10)...(x1)........Failed! After 120 tests.
    [b,c,a]
    {c,[c,a],[c],a,[a,b],[b,a]}
    Old_res ([335,192,185,160,153,123]): true
    New_res: false (why line [1936])
    Shrinking xx.xxxxxx.x.xxxxxxxx.xxxxxxxxxxx(4 times)
    [b,a,c]
 %% {Author1,UPI1,Repair1,Author2,UPI2, Repair2} %%
    {a,      [c], [],     c,      [c,b],[]}
    Old_res ([338,185,160,153,147]): true
    New_res: false (why line [1936])
    false

Old code is wrong: b wasn't repairing in the previous state.

    150> eqc:quickcheck(eqc:testing_time(10, machi_chain_manager1_test:prop_compare_legacy_with_v2_chain_transition_check(whole))).
    xxxxxxxxxxx....x...xxxxx..xx.....x.......xxx..x.......xxx...................x................x......(x10).....(x1)........xFailed! After 130 tests.
    [c,a,b]
    {b,[c],[b,a,c],c,[c,a,b],[b]}
    Old_res ([335,214,185,160,153,147]): true
    New_res: false (why line [1936])
    Shrinking xxxx.x.xxx.xxxxxxx.xxxxxxxxx(4 times)
    [c,b,a]
 %% {Author1,UPI1,Repair1,Author2,UPI2,   Repair2} %%
    {c,      [c], [a,b],  c,      [c,b,a],[]}
    Old_res ([335,328,185,160,153,111]): true
    New_res: false (why line [1981,1679])
    false

Old code is wrong: a & b were repairing but UPI2 has a & b in the wrong order.
 2015-07-04 00:32:28 +09:00
Scott Lystig Fritchie
42fb6dd002 WIP: it's clear that the legacy state transition check is broken, II 2015-07-03 23:37:36 +09:00
Scott Lystig Fritchie
caeb322725 WIP: it's clear that the legacy state transition check is broken 2015-07-03 23:17:34 +09:00
Scott Lystig Fritchie
83015c319d WIP: yeah, now we're going places 2015-07-03 22:05:35 +09:00
Scott Lystig Fritchie
6a706cbfeb WIP: Refactoring and prototyping goop, broken test 2015-07-03 19:21:41 +09:00
Scott Lystig Fritchie
9b3cd9056a Un-TEST'ify testr_react_to_env() everywhere 2015-07-03 16:18:40 +09:00
Scott Lystig Fritchie
78c81f93b7 Make machi_chain_manager1_pulse max commands length longer 2015-07-03 16:06:33 +09:00
Scott Lystig Fritchie
2b64028bbd Add kick_projection_reaction, implement yo:tell_author_yo() 2015-07-03 04:30:05 +09:00
Scott Lystig Fritchie
ff66638eb3 Sequencer changes file sequence number when epoch_id change is detected 2015-07-03 02:04:04 +09:00
Scott Lystig Fritchie
9cf77f4406 WIP: Refactoring and prototyping goop, broken test 2015-07-03 00:59:04 +09:00
Scott Lystig Fritchie
8820a71152 Clean up comment cruft & line wrap yak shaving 2015-07-02 14:44:47 +09:00
Scott Lystig Fritchie
da3a56dd74 Fix epoch checking in eunit tests and enforcement by FLU (always permit list_files()) 2015-07-01 18:12:22 +09:00
Scott Lystig Fritchie
38c1a2ab5d Fix Epoch handling in machi_flu_psup_test.erl 2015-07-01 17:46:35 +09:00
Scott Lystig Fritchie
576d3d76a2 Extend machi_chain_manager1_pulse fudge time factor 2015-07-01 17:46:10 +09:00
Scott Lystig Fritchie
f5ae417b9e Clarify verify_file_checksums_test_ 2015-07-01 14:16:31 +09:00
Scott Lystig Fritchie
670bd2cafc Add some flexibility to machi_chain_manager1_converge_demo:t/1 and t/2 2015-07-01 14:08:17 +09:00
Scott Lystig Fritchie
7542fe8225 WIP: all eunit tests are passing again, yay 2015-06-30 16:12:23 +09:00
Scott Lystig Fritchie
e9d50a2128 WIP: Reinstate one eunit test, fix type bugs 2015-06-30 15:51:03 +09:00
Scott Lystig Fritchie
d54c74f58a WIP: yank out io:format 2015-06-29 16:53:41 +09:00
Scott Lystig Fritchie
3089288338 WIP: giant hairball 13: all unit tests are passing again, yay! 2015-06-29 16:48:06 +09:00
Scott Lystig Fritchie
b25ab3b7ac WIP: giant hairball 11 2015-06-29 16:24:57 +09:00
Scott Lystig Fritchie
64817dd7e8 WIP: giant hairball 01 2015-06-29 16:10:43 +09:00
Scott Lystig Fritchie
3cf18817df WIP: hairball, but timing_pb_encoding_test() works! 2015-06-27 00:12:42 +09:00
Scott Lystig Fritchie
b5c824c5c0 WIP: hairball, but bad_checksum_test() works! 2015-06-27 00:06:21 +09:00
Scott Lystig Fritchie
2fd27fdae6 WIP: hairball, but flu_projection_smoke_test() works! 2015-06-26 23:58:34 +09:00
Scott Lystig Fritchie
920a5c33d7 WIP: giant hairball 6 2015-06-26 22:32:53 +09:00
Scott Lystig Fritchie
77b4da16c3 WIP: giant hairball 5 2015-06-26 21:36:07 +09:00
Scott Lystig Fritchie
6d95d8669c WIP: giant hairball, bleh, low-level checksum_list() barely working 2015-06-26 16:25:12 +09:00
Scott Lystig Fritchie
d9694a992a Alright, use term_to_binary() for opaque/sexp-style encoding, only 15x slower. 
machi_flu1_test: timing_pb_encoding_test_... speed factor=15.12 [2.678 s] ok
 2015-06-25 16:11:46 +09:00
Scott Lystig Fritchie
2763b16ca2 timing_pb_encoding_test_... speed factor=35.95 [2.730 s] ok 
So, the PB style encoding of the Mpb_LL_WriteProjectionReq message
is about 35-36 times slower than using Erlang's term_to_binary()
and binary_to_term().  {sigh}
 2015-06-25 16:11:44 +09:00
Scott Lystig Fritchie
5d8b648a24 All projection store protocol operations are now using Protocol Buffers! 
So, there's some cheating going on, because some of the parts of
the #projection_v1{} and #p_srvr{} records aren't fully specified.
Those parts are being specified as "opaque" in the field names, e.g.

    optional bytes opaque_flap = 10;
    optional bytes opaque_inner = 11;
    required bytes opaque_dbg = 12;
    required bytes opaque_dbg2 = 13;

The serialization that's being used is erlang term sexprs.  That isn't
portable.  So if/when we really need to deal with a non-Erlang
language, we'll have to straighten this out further.
 2015-06-25 15:26:35 +09:00
Scott Lystig Fritchie
4fc0578a9d WIP: bugfixes, machi_flu1_test still broken 2015-06-25 15:08:40 +09:00
Scott Lystig Fritchie
1b0cf06f1c Fix type problem, oops 2015-06-24 14:06:17 +09:00
Scott Lystig Fritchie
2068f70700 WIP: encoding #p_srvr and #projection_v1, just starting. Damn tedious. 2015-06-24 12:50:37 +09:00
Scott Lystig Fritchie
ceebe3d491 WIP: list_files #2 2015-06-23 17:17:14 +09:00
Scott Lystig Fritchie
73f71c406e WIP: list_files end-to-end! 2015-06-23 17:08:15 +09:00
Scott Lystig Fritchie
6722b3c0f1 WIP: checksum_list incomplete implementation.... 2015-06-23 16:53:06 +09:00
Scott Lystig Fritchie
6e77a4ea74 WIP: read_chunk end-to-end! 2015-06-23 16:24:08 +09:00
Scott Lystig Fritchie
44c22bf752 WIP: read_chunk #1 2015-06-23 15:34:48 +09:00
Scott Lystig Fritchie
a8782eed5a WIP: write_chunk #1 2015-06-23 15:13:13 +09:00
Scott Lystig Fritchie
cb06c53dc0 WIP: PB append_chunk end-to-end works! 2015-06-23 14:45:24 +09:00
Scott Lystig Fritchie
5ef499ec73 WIP: append_chunk #1 2015-06-23 14:08:10 +09:00
Scott Lystig Fritchie
bb8e725c26 WIP: 'auth' request placeholders 2015-06-22 18:16:15 +09:00
Scott Lystig Fritchie
db7f1476b9 WIP: 'echo' request works end-to-end, yay! 2015-06-22 18:04:17 +09:00
Scott Lystig Fritchie
3d05f543df WIP: new test case is failing, quick fix soon 2015-06-22 17:49:07 +09:00
Scott Lystig Fritchie
dc9f272c44 Nearly dumbest-possible Protocol Buffers client request & response round trip 2015-06-19 17:21:04 +09:00
Scott Lystig Fritchie
3c300bb9f1 Add write_chunk() to machi_cr_client.erl 2015-06-19 14:49:09 +09:00
Scott Lystig Fritchie
40c0a72b48 Add test/machi_pb_test.erl, finish PB refactoring 2015-06-19 13:00:28 +09:00
Scott Lystig Fritchie
3ce3fb93b9 Use infinity timeout for sanity check 2015-06-17 12:42:53 +09:00
Scott Lystig Fritchie
cc87f682fe Fix broken eunit test machi_flu_psup_test.erl 2015-06-15 13:02:25 +09:00
Scott Lystig Fritchie
b244a3b8e4 Reduce verbosity, try fix up convergence demo for chain len=4 2015-06-15 12:41:16 +09:00
Scott Lystig Fritchie
be62300b3b Bug fixes: model and real bugs, thanks PULSE and converge_demo both! 2015-06-04 17:39:29 +09:00
Scott Lystig Fritchie
d3df2bd31d WIP: remove repair_always_done option, it was flawed 2015-06-03 15:26:22 +09:00
Scott Lystig Fritchie
87417d2872 WIP: get the old jalopy into runnable shape 2015-06-03 11:48:55 +09:00
Scott Lystig Fritchie
c1318d3bbb WIP: wip wip a doowip 2015-06-02 22:13:15 +09:00
Scott Lystig Fritchie
deabe14d29 Un-proplist-ify the inner projection 2015-06-02 20:55:18 +09:00
Scott Lystig Fritchie
207be8729b Un-proplist-ify the flapping_i info 2015-06-02 20:32:52 +09:00
Scott Lystig Fritchie
0f10b45161 Dialyzer fixes, derp! 2015-06-02 19:07:13 +09:00
Scott Lystig Fritchie
67019493aa Round 1 of cleanup 2015-06-02 18:10:45 +09:00
Scott Lystig Fritchie
b51473be09 Change eunit fixture to timeout for machi_cr_client_test:smoke_test 2015-06-02 12:40:07 +09:00
Scott Lystig Fritchie
dd4160b963 Add basic {error, bad_checksum} tests to proxy & CR clients 2015-06-02 12:36:51 +09:00
Scott Lystig Fritchie
e3162fdcda Rudimentary client-side checksum and server-side checksum type tags 2015-06-01 14:25:55 +09:00
Scott Lystig Fritchie
7ba962e9fb Fix handling of {error, partial_read} 2015-05-21 15:12:46 +09:00
Scott Lystig Fritchie
b5ddfaf019 Finish basic API for machi_cr_client.erl 2015-05-19 20:04:36 +09:00
Scott Lystig Fritchie
a4266e8aa4 Fix known chain repair bugs, add basic smoke test 2015-05-19 19:32:48 +09:00
Scott Lystig Fritchie
152e487060 WIP: read-repair, new test is failing, yay 2015-05-19 15:15:05 +09:00
Scott Lystig Fritchie
079d15dd5c Derp, remove debugging goop + fix eunit @ write_chunk() response change 2015-05-19 14:05:18 +09:00
Scott Lystig Fritchie
185c670b2f WIP: refactoring machi_cr_client:append_chunk* 2015-05-18 19:06:06 +09:00
Scott Lystig Fritchie
a06055ac23 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:18:30 +09:00
Scott Lystig Fritchie
6c07522359 Add new API func, append_chunk_extra() 2015-05-17 14:10:42 +09:00
Scott Lystig Fritchie
eec029b08f WIP: aside, fix FLU wedge status @ init() 2015-05-13 17:59:32 +09:00
Scott Lystig Fritchie
0dd9282789 WIP: fix other broken eunit tests, surrounding wedge state 2015-05-08 21:24:07 +09:00
Scott Lystig Fritchie
6f7818fca7 WIP: additional tests for wedge state 2015-05-08 19:50:47 +09:00
Scott Lystig Fritchie
316126fa59 WIP: additional tests for wedge state 2015-05-08 19:07:57 +09:00
Scott Lystig Fritchie
376c4a9ae1 WIP: failing test for append while wedged 2015-05-08 18:22:44 +09:00
Scott Lystig Fritchie
7906e6c235 WIP: basic wedge notifications now working 2015-05-08 18:17:41 +09:00
Scott Lystig Fritchie
ca854373f8 WIP: new test fails: check for not wedged 2015-05-08 17:07:03 +09:00
Scott Lystig Fritchie
d6d003618d WIP: add wedge_status() query to proxy client 2015-05-08 16:58:06 +09:00
Scott Lystig Fritchie
1dc759b908 WIP: add wedge_status() query to client 2015-05-08 16:53:10 +09:00
Scott Lystig Fritchie
762aef557f WIP: Set the stage for FLU wedging API 2015-05-08 15:36:53 +09:00
Scott Lystig Fritchie
ae1d038abe Change default value of chmgr's use_partition_simulator to false 2015-05-08 13:40:44 +09:00
Scott Lystig Fritchie
238c8472cd WIP: timeout comments 2015-05-07 18:52:01 +09:00
Scott Lystig Fritchie
a7bd8e43d3 Clean up machi_flu_psup_test.erl 2015-05-02 17:10:23 +09:00
Scott Lystig Fritchie
1675020150 WIP, tests pass again, includign the newest one 2015-05-02 00:33:49 +09:00
Scott Lystig Fritchie
65993dfcb6 WIP: tests pass, but not finished yet 2015-05-01 14:52:19 +09:00
Scott Lystig Fritchie
53f6a753f4 WIP: tests pass, but not finished yet 2015-05-01 14:51:42 +09:00
Scott Lystig Fritchie
7bafc1c28a WIP: stop for the night, we are broken 2015-04-30 23:16:08 +09:00
Scott Lystig Fritchie
99fd7e7fe1 Add test/machi_flu_psup_test.erl, small cleanups 2015-04-30 21:20:21 +09:00
Scott Lystig Fritchie
442e79e4f1 Add machi_flu_psup.erl to supervise all 3 FLU processes (see below) 
Introduce machi_flu_psup:start_flu_package/4 as a way to start all
related FLU processes
    * The projection store
    * The chain manager
    * The FLU itself

... as well as linked processes.
http://www.snookles.com/scotttmp/flu-tree-20150430.png shows one FLU
running, "a".  The process registered "a" is the append server,
"some-prefix" for the sequencer & writer for the current <<"some-prefix">>
file, and a process each for 3 active TCP connections to that FLU.
 2015-04-30 19:15:27 +09:00
Scott Lystig Fritchie
02bc7fe0bc WIP: Fix bug that flaps inside an inner projection, oops! 2015-04-14 18:23:00 +09:00
Scott Lystig Fritchie
9e587b3d11 WIP: crufty TODO & comment cleanup 2015-04-14 16:17:49 +09:00
Scott Lystig Fritchie
59936eda62 WIP: By Jove, I believe the chain manager is working 2015-04-14 15:30:24 +09:00
Scott Lystig Fritchie
09051aecce WIP: experiments for transitioning out of inner/nested projection state 2015-04-14 00:54:38 +09:00
Scott Lystig Fritchie
0b8ea13f7a WIP: some TODO cleanup & related refactoring 2015-04-10 22:00:52 +09:00
Scott Lystig Fritchie
4334c71a4d WIP: compiler warning fixups 2015-04-10 11:08:17 +09:00
Scott Lystig Fritchie
4f7177067e WIP: Type fixups 2015-04-09 21:32:04 +09:00
Scott Lystig Fritchie
1984c3c350 WIP: convergence demo runs, but badly! 2015-04-09 21:08:15 +09:00
Scott Lystig Fritchie
2b1eb9b144 WIP: Move convergence demo to new module machi_chain_manager1_converg_demo.erl 2015-04-09 18:08:29 +09:00
Scott Lystig Fritchie
6cd9dfc977 WIP: nonunanimous_setup_and_fix_test() passes 2015-04-09 17:47:43 +09:00
Scott Lystig Fritchie
e06adabb6a WIP: bogus flapping in nonunanimous_setup_and_fix_test() 2015-04-09 17:13:38 +09:00
Scott Lystig Fritchie
8deea3bb01 WIP: smoke1 in chain manager works 2015-04-09 14:44:58 +09:00
Scott Lystig Fritchie
a92c5fec0a WIP: remove old smoke0 code from machi_proxy_flu1_client_test.erl 2015-04-09 12:21:15 +09:00
Scott Lystig Fritchie
ce67fb662a WIP: more projection refactoring, eunit tests pass for the moment 2015-04-09 12:16:58 +09:00
Scott Lystig Fritchie
0e38eddaa9 WIP: baby step, machi_chain_manager1_test:smoke0_test() works 2015-04-06 20:07:39 +09:00
Scott Lystig Fritchie
16e283fe5b API overhaul, add machi_proxy_flu1_client.erl, add chain manager (tests commented out) 2015-04-06 18:43:52 +09:00
Scott Lystig Fritchie
a79f385fa7 Fix type problem for return of get_latest_epoch 2015-04-06 15:49:47 +09:00
Scott Lystig Fritchie
99bfa2a3b8 Import of machi_chain_manager1.erl and friends; tests broken 2015-04-06 14:16:20 +09:00
Scott Lystig Fritchie
c27aa1f579 Projection store API complete, I think 2015-04-03 18:37:09 +09:00
Scott Lystig Fritchie
022b9c4d1f WIP: projection store: read, get latest epoch 2015-04-03 17:55:35 +09:00
Scott Lystig Fritchie
acf54e3c21 WIP: client side projection store, 1st API op (write), part II 2015-04-03 17:10:52 +09:00
Scott Lystig Fritchie
7205c5283e WIP: client side projection store, 1st API op (write) 2015-04-03 12:36:16 +09:00
Scott Lystig Fritchie
7627ba08a3 WIP: epoch ID added to write/delete/trunc protocol commands 2015-04-02 21:18:41 +09:00
Scott Lystig Fritchie
3aaa2c3a3d WIP: epoch ID added to list protocol command 2015-04-02 21:01:48 +09:00
Scott Lystig Fritchie
6b8a3cf2a4 WIP: epoch ID added to checksum protocol command 2015-04-02 20:49:45 +09:00
Scott Lystig Fritchie
9479baac46 WIP: epoch ID added to read protocol command 2015-04-02 20:31:10 +09:00
Scott Lystig Fritchie
44bb5e1dae WIP: epoch ID added to append protocol command 2015-04-02 18:08:42 +09:00
Scott Lystig Fritchie
5580098d49 Refactor to use record for FLU state, add dbg mode 2015-04-02 17:16:15 +09:00
Scott Lystig Fritchie
4c3bd81689 Add machi_projection.erl and basic new() test 2015-04-02 16:24:32 +09:00
Scott Lystig Fritchie
f8263c15cc Move FLU client 'verify checksums + local path' code from prototype/demo-day-hack 2015-04-02 12:38:12 +09:00
Scott Lystig Fritchie
76fcd4d931 Move FLU client 'verify checksums' code from prototype/demo-day-hack 2015-04-01 18:35:10 +09:00
Scott Lystig Fritchie
5c20ee6337 Fix client API for file list & checksum list 2015-04-01 18:02:16 +09:00
Scott Lystig Fritchie
d243ffca23 Single server client & server code (squashed) 2015-04-01 16:14:24 +09:00