greg/machi
1
0
Fork 0
Code Issues 11 Pull requests 1 Projects Releases Packages Wiki Activity Actions
622 commits 58 branches 0 tags 5.9 MiB
Commit graph

240 commits

Author SHA1 Message Date
Scott Lystig Fritchie
83015c319d WIP: yeah, now we're going places 2015-07-03 22:05:35 +09:00
Scott Lystig Fritchie
6a706cbfeb WIP: Refactoring and prototyping goop, broken test 2015-07-03 19:21:41 +09:00
Scott Lystig Fritchie
9b3cd9056a Un-TEST'ify testr_react_to_env() everywhere 2015-07-03 16:18:40 +09:00
Scott Lystig Fritchie
2b64028bbd Add kick_projection_reaction, implement yo:tell_author_yo() 2015-07-03 04:30:05 +09:00
Scott Lystig Fritchie
c6870a1c86 If FLU is wedged by a newer client epoch ID, kick the chain manager to react 2015-07-03 02:17:01 +09:00
Scott Lystig Fritchie
ff66638eb3 Sequencer changes file sequence number when epoch_id change is detected 2015-07-03 02:04:04 +09:00
Scott Lystig Fritchie
da3a56dd74 Fix epoch checking in eunit tests and enforcement by FLU (always permit list_files()) 2015-07-01 18:12:22 +09:00
Scott Lystig Fritchie
2c869ed598 TODO fix: wedge self 2015-07-01 17:19:11 +09:00
Scott Lystig Fritchie
1e14fe878f Ha, oops! Add bad_epoch code, derp 1 2015-07-01 15:51:25 +09:00
Scott Lystig Fritchie
a658a64482 Cosmetic formatting change 2015-07-01 15:37:53 +09:00
Scott Lystig Fritchie
a0061d6ffa make decode_csum_file_entry() very slightly less brittle 2015-07-01 15:18:57 +09:00
Scott Lystig Fritchie
d710d90ea7 Fix usage of checksum_list by machi_chain_repair.erl 2015-07-01 15:04:22 +09:00
Scott Lystig Fritchie
0321e05b46 Fix usage of checksum_list by machi_basho_bench_driver.erl 2015-07-01 15:03:56 +09:00
Scott Lystig Fritchie
e3b80c6ac2 Docuemntation updates 2015-06-30 19:04:23 +09:00
Scott Lystig Fritchie
00c8cf0ef7 Rename temporary HTTP server hack functions 2015-06-30 16:19:44 +09:00
Scott Lystig Fritchie
7542fe8225 WIP: all eunit tests are passing again, yay 2015-06-30 16:12:23 +09:00
Scott Lystig Fritchie
e9d50a2128 WIP: Reinstate one eunit test, fix type bugs 2015-06-30 15:51:03 +09:00
Scott Lystig Fritchie
3d2b49b7e5 WIP: refactoring & edoc'ing 2015-06-30 15:20:35 +09:00
Scott Lystig Fritchie
310fdb1f6a Add crude file size check to do_server_checksum_listing() 2015-06-30 14:13:26 +09:00
Scott Lystig Fritchie
2d070bf1e3 Minor refactoring + add demo/exploratory time measurement code 
%% Demo/exploratory hackery to check relative speeds of dealing with
%% checksum data in different ways.
%%
%% Summary:
%%
%% * Use compact binary encoding, with 1 byte header for entry length.
%%     * Because the hex-style code is *far* slower just for enc & dec ops.
%%     * For 1M entries of enc+dec: 0.215 sec vs. 15.5 sec.
%% * File sorter when sorting binaries as-is is only 30-40% slower
%%   than an in-memory split (of huge binary emulated by file:read_file()
%%   "big slurp") and sort of the same as-is sortable binaries.
%% * File sorter slows by a factor of about 2.5 if {order, fun compare/2}
%%   function must be used, i.e. because the checksum entry lengths differ.
%% * File sorter + {order, fun compare/2} is still *far* faster than external
%%   sort by OS X's sort(1) of sortable ASCII hex-style:
%%   4.5 sec vs. 21 sec.
%% * File sorter {order, fun compare/2} is faster than in-memory sort
%%   of order-friendly 3-tuple-style: 4.5 sec vs. 15 sec.
 2015-06-30 14:08:46 +09:00
Scott Lystig Fritchie
34b046acbd Remove machi_pb_wrap.erl 2015-06-29 17:31:07 +09:00
Scott Lystig Fritchie
dba7041929 Change names to indicate we're no longer in PB land 2015-06-29 17:20:17 +09:00
Scott Lystig Fritchie
151e696324 WIP: yank out more unused cruft 2015-06-29 17:14:33 +09:00
Scott Lystig Fritchie
87ec988353 WIP: yank out more unused cruft 2015-06-29 17:06:28 +09:00
Scott Lystig Fritchie
6cd3b8d0ec WIP: yank out lots of unused cruft 2015-06-29 17:02:58 +09:00
Scott Lystig Fritchie
d54c74f58a WIP: yank out io:format 2015-06-29 16:53:41 +09:00
Scott Lystig Fritchie
7aff9fca70 WIP: giant hairball 12 2015-06-29 16:42:05 +09:00
Scott Lystig Fritchie
b25ab3b7ac WIP: giant hairball 11 2015-06-29 16:24:57 +09:00
Scott Lystig Fritchie
64817dd7e8 WIP: giant hairball 01 2015-06-29 16:10:43 +09:00
Scott Lystig Fritchie
f45dc7829e WIP: hairball, but: Failed: 6. Skipped: 0. Passed: 13 2015-06-27 00:43:27 +09:00
Scott Lystig Fritchie
b5c824c5c0 WIP: hairball, but bad_checksum_test() works! 2015-06-27 00:06:21 +09:00
Scott Lystig Fritchie
2fd27fdae6 WIP: hairball, but flu_projection_smoke_test() works! 2015-06-26 23:58:34 +09:00
Scott Lystig Fritchie
93f64a20c0 WIP: hairball, but flu_smoke_test() works! 2015-06-26 23:03:28 +09:00
Scott Lystig Fritchie
920a5c33d7 WIP: giant hairball 6 2015-06-26 22:32:53 +09:00
Scott Lystig Fritchie
77b4da16c3 WIP: giant hairball 5 2015-06-26 21:36:07 +09:00
Scott Lystig Fritchie
9a212fb19f WIP: giant hairball 4 2015-06-26 20:47:55 +09:00
Scott Lystig Fritchie
0e32fd25c9 WIP: giant hairball 3 2015-06-26 18:59:07 +09:00
Scott Lystig Fritchie
8437d76c1c WIP: giant hairball 2 2015-06-26 18:22:15 +09:00
Scott Lystig Fritchie
fb975eea46 WIP: giant hairball 2015-06-26 16:58:24 +09:00
Scott Lystig Fritchie
6d95d8669c WIP: giant hairball, bleh, low-level checksum_list() barely working 2015-06-26 16:25:12 +09:00
Scott Lystig Fritchie
90efc41167 machi.proto definition for low-level protocol ops 2015-06-25 17:09:33 +09:00
Scott Lystig Fritchie
cf0d9a25b4 EDoc cleanup 2015-06-25 16:39:19 +09:00
Scott Lystig Fritchie
0b2b79cd0b Merge branch 'slf/pb-api-experiment1' 2015-06-25 16:36:50 +09:00
Scott Lystig Fritchie
0f4d5ed775 Silence dialyzer unused function clause 2015-06-25 16:36:29 +09:00
Scott Lystig Fritchie
c2faf9f499 yolo, un-do experimental type hack 2015-06-25 16:36:14 +09:00
Scott Lystig Fritchie
d9694a992a Alright, use term_to_binary() for opaque/sexp-style encoding, only 15x slower. 
machi_flu1_test: timing_pb_encoding_test_... speed factor=15.12 [2.678 s] ok
 2015-06-25 16:11:46 +09:00
Scott Lystig Fritchie
2763b16ca2 timing_pb_encoding_test_... speed factor=35.95 [2.730 s] ok 
So, the PB style encoding of the Mpb_LL_WriteProjectionReq message
is about 35-36 times slower than using Erlang's term_to_binary()
and binary_to_term().  {sigh}
 2015-06-25 16:11:44 +09:00
Scott Lystig Fritchie
5d8b648a24 All projection store protocol operations are now using Protocol Buffers! 
So, there's some cheating going on, because some of the parts of
the #projection_v1{} and #p_srvr{} records aren't fully specified.
Those parts are being specified as "opaque" in the field names, e.g.

    optional bytes opaque_flap = 10;
    optional bytes opaque_inner = 11;
    required bytes opaque_dbg = 12;
    required bytes opaque_dbg2 = 13;

The serialization that's being used is erlang term sexprs.  That isn't
portable.  So if/when we really need to deal with a non-Erlang
language, we'll have to straighten this out further.
 2015-06-25 15:26:35 +09:00
Scott Lystig Fritchie
841235b3b5 WIP: bugfixes, add {error, written} 2015-06-25 15:10:24 +09:00
Scott Lystig Fritchie
4fc0578a9d WIP: bugfixes, machi_flu1_test still broken 2015-06-25 15:08:40 +09:00
Scott Lystig Fritchie
d9407b76b7 WIP: dinnertime, machi_flu1_test still broken 2015-06-24 18:00:25 +09:00
Scott Lystig Fritchie
31c5bcc0c7 WIP: 1/2 of low-level projection proto finished, machi_flu1_test fails 2015-06-24 17:20:18 +09:00
Scott Lystig Fritchie
725b10ba90 Complete PB round-trip for #projection_v1{}, bleh 2015-06-24 16:13:11 +09:00
Scott Lystig Fritchie
2068f70700 WIP: encoding #p_srvr and #projection_v1, just starting. Damn tedious. 2015-06-24 12:50:37 +09:00
Scott Lystig Fritchie
817efb2b15 machi_pb_high_client: always be checksumming 2015-06-23 17:37:47 +09:00
Scott Lystig Fritchie
d3b0b7fdc5 Clean up some dialyzer complaints 2015-06-23 17:26:15 +09:00
Scott Lystig Fritchie
727b2a987d ROTFL forgot to add src/machi_pb_server.erl 2015-06-23 17:22:45 +09:00
Scott Lystig Fritchie
ceebe3d491 WIP: list_files #2 2015-06-23 17:17:14 +09:00
Scott Lystig Fritchie
73f71c406e WIP: list_files end-to-end! 2015-06-23 17:08:15 +09:00
Scott Lystig Fritchie
6722b3c0f1 WIP: checksum_list incomplete implementation.... 2015-06-23 16:53:06 +09:00
Scott Lystig Fritchie
44c22bf752 WIP: read_chunk #1 2015-06-23 15:34:48 +09:00
Scott Lystig Fritchie
a8782eed5a WIP: write_chunk #1 2015-06-23 15:13:13 +09:00
Scott Lystig Fritchie
cb06c53dc0 WIP: PB append_chunk end-to-end works! 2015-06-23 14:45:24 +09:00
Scott Lystig Fritchie
5ef499ec73 WIP: append_chunk #1 2015-06-23 14:08:10 +09:00
Scott Lystig Fritchie
bb8e725c26 WIP: 'auth' request placeholders 2015-06-22 18:16:15 +09:00
Scott Lystig Fritchie
db7f1476b9 WIP: 'echo' request works end-to-end, yay! 2015-06-22 18:04:17 +09:00
Scott Lystig Fritchie
3d05f543df WIP: new test case is failing, quick fix soon 2015-06-22 17:49:07 +09:00
Scott Lystig Fritchie
dc9f272c44 Nearly dumbest-possible Protocol Buffers client request & response round trip 2015-06-19 17:21:04 +09:00
Scott Lystig Fritchie
c4bdeee4da Oops, add missing src/machi_dt.erl 2015-06-19 17:20:49 +09:00
Scott Lystig Fritchie
0cdaee32f8 Egadz, edoc doesn't use preprocessor {sigh} 2015-06-19 16:24:57 +09:00
Scott Lystig Fritchie
984b4f7a86 Dialyzer tightening and subsequent cleanup 2015-06-19 16:04:34 +09:00
Scott Lystig Fritchie
1372bd9594 {sigh} add filter-dialyzer-dep-warnings 2015-06-19 15:22:07 +09:00
Scott Lystig Fritchie
3c300bb9f1 Add write_chunk() to machi_cr_client.erl 2015-06-19 14:49:09 +09:00
Scott Lystig Fritchie
40c0a72b48 Add test/machi_pb_test.erl, finish PB refactoring 2015-06-19 13:00:28 +09:00
Scott Lystig Fritchie
a82bd68f3c Overhaul the 0.1 PB definition. Again. 
Many thanks to @seancribbs for a suggestion to avoid the PB design
mistake/feature of the original Riak KV PB API.
 2015-06-19 12:28:31 +09:00
Scott Lystig Fritchie
87b636a349 WIP: PB wrestling 2015-06-18 17:31:48 +09:00
Scott Lystig Fritchie
37a8c1c124 WIP: PB wrestling 2015-06-18 16:16:23 +09:00
Scott Lystig Fritchie
e5673b5e20 First attempt at Protocol Buffers .proto + infrastructure 2015-06-17 16:12:20 +09:00
Scott Lystig Fritchie
22337e1819 Remove short circuit (bad idea!) from react_to_env_C100() 2015-06-15 17:22:02 +09:00
Scott Lystig Fritchie
b244a3b8e4 Reduce verbosity, try fix up convergence demo for chain len=4 2015-06-15 12:41:16 +09:00
Scott Lystig Fritchie
9bf76e0bfb Fix for correctness bug, thanks PULSE 2015-06-05 01:06:39 +09:00
Scott Lystig Fritchie
be62300b3b Bug fixes: model and real bugs, thanks PULSE and converge_demo both! 2015-06-04 17:39:29 +09:00
Scott Lystig Fritchie
0cf9627f26 Bugfix, found by inspection, yay! 2015-06-04 15:05:37 +09:00
Scott Lystig Fritchie
89b8b6a012 Bugfix, found by PULSE, yay! 2015-06-04 14:31:58 +09:00
Scott Lystig Fritchie
d3df2bd31d WIP: remove repair_always_done option, it was flawed 2015-06-03 15:26:22 +09:00
Scott Lystig Fritchie
87417d2872 WIP: get the old jalopy into runnable shape 2015-06-03 11:48:55 +09:00
Scott Lystig Fritchie
2207151eba Fix projection_transition_is_sane() bug 2015-06-02 21:20:50 +09:00
Scott Lystig Fritchie
deabe14d29 Un-proplist-ify the inner projection 2015-06-02 20:55:18 +09:00
Scott Lystig Fritchie
207be8729b Un-proplist-ify the flapping_i info 2015-06-02 20:32:52 +09:00
Scott Lystig Fritchie
0f10b45161 Dialyzer fixes, derp! 2015-06-02 19:07:13 +09:00
Scott Lystig Fritchie
000d687588 Fix creation_time bug in inner projection 2015-06-02 16:26:49 +09:00
Scott Lystig Fritchie
cffbd3c50c Add checksum handling strawman to strawman HTTP interface 2015-06-02 13:23:36 +09:00
Scott Lystig Fritchie
dd4160b963 Add basic {error, bad_checksum} tests to proxy & CR clients 2015-06-02 12:36:51 +09:00
Scott Lystig Fritchie
e3162fdcda Rudimentary client-side checksum and server-side checksum type tags 2015-06-01 14:25:55 +09:00
Scott Lystig Fritchie
6cebf39723 Damn ugly HTTP interface "equivalent" for machi_cr_client.erl basic API 
This goes to show that mixing implementation and protocol and API
and lots of other stuff ... is cool for the quick hack to do one thing
but really sucks when trying to do more than one thing.

* Proof-of-concept only: add HTTP/1.0'ish 'PUT' interface to be the
rough equivalent of machi_cr_client:append_chunk/3
* Proof-of-concept only: add HTTP/1.0'ish 'GET' interface to be the
rough equivalent of machi_cr_client:read_chunk/4

Example use: `append_chunk`

    % curl http://127.0.0.1:4444/foo -0 -T /etc/hosts -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > PUT /foo HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    > Content-Length: 338
    >
    * We are completely uploaded and fine
    * HTTP 1.0, assume close after body
    < HTTP/1.0 201 Created
    < Location: foo.50EI18AX.21
    < X-Offset: 3052
    < X-Size: 338
    <
    * Closing connection 0

Example_use: `read_chunk`

    curl 'http://127.0.0.1:4444/foo.50EI18AX.21?offset=3052&size=338' -0 -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > GET /foo.50EI18AX.21?offset=3052&size=338 HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    >
    * HTTP 1.0, assume close after body
    < HTTP/1.0 200 OK
    < Content-Length: 338
    <
    ##
    # Host Database
    #
    # localhost is used to configure the loopback interface
    # when the system is booting.  Do not change this entry.
    ##
    127.0.0.1	localhost
    127.0.0.1	test.localhost
    255.255.255.255	broadcasthost
    ::1             localhost
    fe80::1%lo0	localhost

    # Xxxxxxx Yyyyy
    192.168.99.222	zzzzz

    127.0.0.1   aaaaaaaa.bb.ccccccccc.com
    * Closing connection 0
 2015-05-22 17:51:06 +09:00
Scott Lystig Fritchie
7ba962e9fb Fix handling of {error, partial_read} 2015-05-21 15:12:46 +09:00
Scott Lystig Fritchie
9e41162e65 Minor machi_basho_bench_driver changes 2015-05-20 18:44:04 +09:00
Scott Lystig Fritchie
1c13273efd Add simple basho_bench driver, no schmancy bells or whistles 2015-05-20 17:30:37 +09:00
Scott Lystig Fritchie
69244691f4 Such wonder when one *reads* the docs... 2015-05-20 14:12:48 +09:00
Scott Lystig Fritchie
b5ddfaf019 Finish basic API for machi_cr_client.erl 2015-05-19 20:04:36 +09:00