greg/machi
1
0
Fork 0
Code Issues 11 Pull requests 1 Projects Releases Packages Wiki Activity Actions
924 commits 58 branches 0 tags 5.9 MiB
Commit graph

448 commits

Author SHA1 Message Date
Scott Lystig Fritchie
dd4160b963 Add basic {error, bad_checksum} tests to proxy & CR clients 2015-06-02 12:36:51 +09:00
Scott Lystig Fritchie
e3162fdcda Rudimentary client-side checksum and server-side checksum type tags 2015-06-01 14:25:55 +09:00
Scott Lystig Fritchie
6cebf39723 Damn ugly HTTP interface "equivalent" for machi_cr_client.erl basic API 
This goes to show that mixing implementation and protocol and API
and lots of other stuff ... is cool for the quick hack to do one thing
but really sucks when trying to do more than one thing.

* Proof-of-concept only: add HTTP/1.0'ish 'PUT' interface to be the
rough equivalent of machi_cr_client:append_chunk/3
* Proof-of-concept only: add HTTP/1.0'ish 'GET' interface to be the
rough equivalent of machi_cr_client:read_chunk/4

Example use: `append_chunk`

    % curl http://127.0.0.1:4444/foo -0 -T /etc/hosts -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > PUT /foo HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    > Content-Length: 338
    >
    * We are completely uploaded and fine
    * HTTP 1.0, assume close after body
    < HTTP/1.0 201 Created
    < Location: foo.50EI18AX.21
    < X-Offset: 3052
    < X-Size: 338
    <
    * Closing connection 0

Example_use: `read_chunk`

    curl 'http://127.0.0.1:4444/foo.50EI18AX.21?offset=3052&size=338' -0 -v
    * Hostname was NOT found in DNS cache
    *   Trying 127.0.0.1...
    * Connected to 127.0.0.1 (127.0.0.1) port 4444 (#0)
    > GET /foo.50EI18AX.21?offset=3052&size=338 HTTP/1.0
    > User-Agent: curl/7.37.1
    > Host: 127.0.0.1:4444
    > Accept: */*
    >
    * HTTP 1.0, assume close after body
    < HTTP/1.0 200 OK
    < Content-Length: 338
    <
    ##
    # Host Database
    #
    # localhost is used to configure the loopback interface
    # when the system is booting.  Do not change this entry.
    ##
    127.0.0.1	localhost
    127.0.0.1	test.localhost
    255.255.255.255	broadcasthost
    ::1             localhost
    fe80::1%lo0	localhost

    # Xxxxxxx Yyyyy
    192.168.99.222	zzzzz

    127.0.0.1   aaaaaaaa.bb.ccccccccc.com
    * Closing connection 0
 2015-05-22 17:51:06 +09:00
Scott Lystig Fritchie
7ba962e9fb Fix handling of {error, partial_read} 2015-05-21 15:12:46 +09:00
Scott Lystig Fritchie
9e41162e65 Minor machi_basho_bench_driver changes 2015-05-20 18:44:04 +09:00
Scott Lystig Fritchie
1c13273efd Add simple basho_bench driver, no schmancy bells or whistles 2015-05-20 17:30:37 +09:00
Scott Lystig Fritchie
69244691f4 Such wonder when one *reads* the docs... 2015-05-20 14:12:48 +09:00
Scott Lystig Fritchie
b5ddfaf019 Finish basic API for machi_cr_client.erl 2015-05-19 20:04:36 +09:00
Scott Lystig Fritchie
a4266e8aa4 Fix known chain repair bugs, add basic smoke test 2015-05-19 19:32:48 +09:00
Scott Lystig Fritchie
152e487060 WIP: read-repair, new test is failing, yay 2015-05-19 15:15:05 +09:00
Scott Lystig Fritchie
079d15dd5c Derp, remove debugging goop + fix eunit @ write_chunk() response change 2015-05-19 14:05:18 +09:00
Scott Lystig Fritchie
eaf007ec08 Fix read repair FLU tracking 2015-05-19 13:56:12 +09:00
Scott Lystig Fritchie
f7274e7106 WIP: brute-force read-repair 2015-05-18 23:26:21 +09:00
Scott Lystig Fritchie
185c670b2f WIP: refactoring machi_cr_client:append_chunk* 2015-05-18 19:06:06 +09:00
Scott Lystig Fritchie
a347722a15 Fix {error,not_written} type bugs in chmgr 2015-05-18 17:32:22 +09:00
Scott Lystig Fritchie
966d2edae8 WIP: refactoring machi_cr_client:append_chunk* 2015-05-18 15:49:05 +09:00
Scott Lystig Fritchie
a7f53cf21a WIP: starting machi_cr_client:append_chunk* 2015-05-18 00:59:24 +09:00
Scott Lystig Fritchie
b0607ae815 WIP: starting machi_cr_client:append_chunk* 2015-05-18 00:33:15 +09:00
Scott Lystig Fritchie
d293170e92 WIP: starting machi_cr_client.erl 2015-05-17 23:48:05 +09:00
Scott Lystig Fritchie
10364834de Add a dummy client-side implementation module:machi_yessir_client.erl 2015-05-17 19:00:51 +09:00
Scott Lystig Fritchie
d4080b78d8 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:46:50 +09:00
Scott Lystig Fritchie
a8c5879d21 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:25:58 +09:00
Scott Lystig Fritchie
a06055ac23 WIP: rearrange client code to approach some semblance of modularity someday 2015-05-17 16:18:30 +09:00
Scott Lystig Fritchie
c7d4131a44 Remove unused verb() 2015-05-17 14:19:37 +09:00
Scott Lystig Fritchie
6c07522359 Add new API func, append_chunk_extra() 2015-05-17 14:10:42 +09:00
Scott Lystig Fritchie
5c2635346f Basic multi-party chain repair for ap_mode finished 2015-05-16 17:39:58 +09:00
Scott Lystig Fritchie
d2f1549aa3 WIP: more generic all-way file chunk merge func, part 3 2015-05-16 17:11:54 +09:00
Scott Lystig Fritchie
04bc28b9da WIP: more generic all-way file chunk merge func, part 2 2015-05-16 16:55:48 +09:00
Scott Lystig Fritchie
a9c753ad64 WIP: more generic all-way file chunk merge func 2015-05-15 17:15:02 +09:00
Scott Lystig Fritchie
358764d403 WIP: repair common file, part 0 2015-05-14 14:04:31 +09:00
Scott Lystig Fritchie
19d3c95325 WIP: aside, damn, add missing hex encoding for epochid, derp 2015-05-13 18:57:38 +09:00
Scott Lystig Fritchie
eec029b08f WIP: aside, fix FLU wedge status @ init() 2015-05-13 17:59:32 +09:00
Scott Lystig Fritchie
4ae0f94649 WIP: move to stats via ETS, success/failure propagates, yay! 2015-05-12 23:45:35 +09:00
Scott Lystig Fritchie
cad84442bb WIP: stats record, hrm 2015-05-12 22:42:03 +09:00
Scott Lystig Fritchie
8807f954ff WIP: Whole file repair is 95% complete, yay! 2015-05-12 21:45:40 +09:00
Scott Lystig Fritchie
f48720e4dc WIP: set up proxies for repair 2015-05-12 12:56:41 +09:00
Scott Lystig Fritchie
1c70a46b09 Add basic process & bookkeeping structure for repair proc 
=INFO REPORT==== 11-May-2015::19:50:09 ===
    Chain tail a of [a] starting repair of [c]

    =INFO REPORT==== 11-May-2015::19:50:12 ===
    Chain tail a of [a]: repair finished in 2.438 seconds: todo_yo
 2015-05-11 19:50:21 +09:00
Scott Lystig Fritchie
c82000dc30 Reduce spamminess slightly 2015-05-11 19:00:21 +09:00
Scott Lystig Fritchie
33bfbe109e Chain manager bug fixes & enhancment (more...) 
* Set max length of a chain at -define(MAX_CHAIN_LENGTH, 64).

* Perturb tick sleep time of each manager

* If a chain manager L has zero members in its chain, and then its local
public projection store (authored by some remote author R) has a projection
that contains L, then adopt R's projection and start humming consensus.

* Handle "cross-talk" across projection stores, when chain membership
is changed administratively, e.g. chain was [a,b,c] then changed to merely
[a], but that change only happens on a.  Servers b & c continue to use
stale projections and scribble their projection suggestions to a, causing
it to flap.

What's really cool about the flapping handling is that it *works*.  I
wasn't thinking about this scenario when designing the flapping logic, but
it's really nifty that this extra scenario causes a to flap and then a's
inner projection remains stable, yay!

* Add complaints when "cross-talk" is observed.

* Fix flapping sleep time throttle.

* Fix bug in the machi_projection_store.erl's bookkeeping of the
max epoch number when flapping.
 2015-05-11 18:41:45 +09:00
Scott Lystig Fritchie
dc43a32945 WIP: tests for wedge state all working 2015-05-08 21:37:19 +09:00
Scott Lystig Fritchie
0dd9282789 WIP: fix other broken eunit tests, surrounding wedge state 2015-05-08 21:24:07 +09:00
Scott Lystig Fritchie
6f7818fca7 WIP: additional tests for wedge state 2015-05-08 19:50:47 +09:00
Scott Lystig Fritchie
316126fa59 WIP: additional tests for wedge state 2015-05-08 19:07:57 +09:00
Scott Lystig Fritchie
7906e6c235 WIP: basic wedge notifications now working 2015-05-08 18:17:41 +09:00
Scott Lystig Fritchie
d6d003618d WIP: add wedge_status() query to proxy client 2015-05-08 16:58:06 +09:00
Scott Lystig Fritchie
1dc759b908 WIP: add wedge_status() query to client 2015-05-08 16:53:10 +09:00
Scott Lystig Fritchie
762aef557f WIP: Set the stage for FLU wedging API 2015-05-08 15:36:53 +09:00
Scott Lystig Fritchie
ae1d038abe Change default value of chmgr's use_partition_simulator to false 2015-05-08 13:40:44 +09:00
Scott Lystig Fritchie
238c8472cd WIP: timeout comments 2015-05-07 18:52:01 +09:00
Scott Lystig Fritchie
14fc37bd0d Add ability to start FLUs at application startup 2015-05-07 18:39:39 +09:00
Scott Lystig Fritchie
517941aaaa Finish chain manager restart & membership changing 2015-05-07 17:52:16 +09:00
Scott Lystig Fritchie
aeb2e4ef9e WIP: partial refactoring of chmgr 2nd start code, one test broken 2015-05-06 11:41:04 +09:00
Scott Lystig Fritchie
a7bd8e43d3 Clean up machi_flu_psup_test.erl 2015-05-02 17:10:23 +09:00
Scott Lystig Fritchie
1675020150 WIP, tests pass again, includign the newest one 2015-05-02 00:33:49 +09:00
Scott Lystig Fritchie
53f6a753f4 WIP: tests pass, but not finished yet 2015-05-01 14:51:42 +09:00
Scott Lystig Fritchie
7bafc1c28a WIP: stop for the night, we are broken 2015-04-30 23:16:08 +09:00
Scott Lystig Fritchie
99fd7e7fe1 Add test/machi_flu_psup_test.erl, small cleanups 2015-04-30 21:20:21 +09:00
Scott Lystig Fritchie
442e79e4f1 Add machi_flu_psup.erl to supervise all 3 FLU processes (see below) 
Introduce machi_flu_psup:start_flu_package/4 as a way to start all
related FLU processes
    * The projection store
    * The chain manager
    * The FLU itself

... as well as linked processes.
http://www.snookles.com/scotttmp/flu-tree-20150430.png shows one FLU
running, "a".  The process registered "a" is the append server,
"some-prefix" for the sequencer & writer for the current <<"some-prefix">>
file, and a process each for 3 active TCP connections to that FLU.
 2015-04-30 19:15:27 +09:00
Scott Lystig Fritchie
9249663ff6 TODO updates for machi_flu1.erl based on doc rewriting work 2015-04-30 15:15:43 +09:00
Scott Lystig Fritchie
02bc7fe0bc WIP: Fix bug that flaps inside an inner projection, oops! 2015-04-14 18:23:00 +09:00
Scott Lystig Fritchie
90df655256 WIP: Ha! There's a bug, this verbose logging change made it easier to see 2015-04-14 16:38:19 +09:00
Scott Lystig Fritchie
9e587b3d11 WIP: crufty TODO & comment cleanup 2015-04-14 16:17:49 +09:00
Scott Lystig Fritchie
59936eda62 WIP: By Jove, I believe the chain manager is working 2015-04-14 15:30:24 +09:00
Scott Lystig Fritchie
09051aecce WIP: experiments for transitioning out of inner/nested projection state 2015-04-14 00:54:38 +09:00
Scott Lystig Fritchie
94298d90da WIP: transitions into & out of inner proj nesting are problems, yo! 2015-04-10 22:41:22 +09:00
Scott Lystig Fritchie
0b8ea13f7a WIP: some TODO cleanup & related refactoring 2015-04-10 22:00:52 +09:00
Scott Lystig Fritchie
876bf79835 Add debugging & TODO note about using inner projection 2015-04-10 14:15:16 +09:00
Scott Lystig Fritchie
4f7177067e WIP: Type fixups 2015-04-09 21:32:04 +09:00
Scott Lystig Fritchie
1984c3c350 WIP: convergence demo runs, but badly! 2015-04-09 21:08:15 +09:00
Scott Lystig Fritchie
6cd9dfc977 WIP: nonunanimous_setup_and_fix_test() passes 2015-04-09 17:47:43 +09:00
Scott Lystig Fritchie
e06adabb6a WIP: bogus flapping in nonunanimous_setup_and_fix_test() 2015-04-09 17:13:38 +09:00
Scott Lystig Fritchie
8deea3bb01 WIP: smoke1 in chain manager works 2015-04-09 14:44:58 +09:00
Scott Lystig Fritchie
a92c5fec0a WIP: remove old smoke0 code from machi_proxy_flu1_client_test.erl 2015-04-09 12:21:15 +09:00
Scott Lystig Fritchie
ce67fb662a WIP: more projection refactoring, eunit tests pass for the moment 2015-04-09 12:16:58 +09:00
Scott Lystig Fritchie
0b2866d102 Add -spec statements to machi_util.erl, clean up the fallout 2015-04-08 18:39:55 +09:00
Scott Lystig Fritchie
ad872e23ca Add first basic round of EDoc documentation, 'make edoc' target 2015-04-08 17:32:01 +09:00
Scott Lystig Fritchie
0e38eddaa9 WIP: baby step, machi_chain_manager1_test:smoke0_test() works 2015-04-06 20:07:39 +09:00
Scott Lystig Fritchie
16e283fe5b API overhaul, add machi_proxy_flu1_client.erl, add chain manager (tests commented out) 2015-04-06 18:43:52 +09:00
Scott Lystig Fritchie
1d63b93fc0 Kill append, projection, and listen pids on machi_flu1:stop() 2015-04-06 16:49:17 +09:00
Scott Lystig Fritchie
a79f385fa7 Fix type problem for return of get_latest_epoch 2015-04-06 15:49:47 +09:00
Scott Lystig Fritchie
99bfa2a3b8 Import of machi_chain_manager1.erl and friends; tests broken 2015-04-06 14:16:20 +09:00
Scott Lystig Fritchie
c27aa1f579 Projection store API complete, I think 2015-04-03 18:37:09 +09:00
Scott Lystig Fritchie
022b9c4d1f WIP: projection store: read, get latest epoch 2015-04-03 17:55:35 +09:00
Scott Lystig Fritchie
acf54e3c21 WIP: client side projection store, 1st API op (write), part II 2015-04-03 17:10:52 +09:00
Scott Lystig Fritchie
7205c5283e WIP: client side projection store, 1st API op (write) 2015-04-03 12:36:16 +09:00
Scott Lystig Fritchie
7627ba08a3 WIP: epoch ID added to write/delete/trunc protocol commands 2015-04-02 21:18:41 +09:00
Scott Lystig Fritchie
3aaa2c3a3d WIP: epoch ID added to list protocol command 2015-04-02 21:01:48 +09:00
Scott Lystig Fritchie
6b8a3cf2a4 WIP: epoch ID added to checksum protocol command 2015-04-02 20:49:45 +09:00
Scott Lystig Fritchie
9479baac46 WIP: epoch ID added to read protocol command 2015-04-02 20:31:10 +09:00
Scott Lystig Fritchie
44bb5e1dae WIP: epoch ID added to append protocol command 2015-04-02 18:08:42 +09:00
Scott Lystig Fritchie
030d2ecd10 Update TODO-shortterm.org + minor stuff 2015-04-02 17:42:26 +09:00
Scott Lystig Fritchie
5580098d49 Refactor to use record for FLU state, add dbg mode 2015-04-02 17:16:15 +09:00
Scott Lystig Fritchie
4c3bd81689 Add machi_projection.erl and basic new() test 2015-04-02 16:24:32 +09:00
Scott Lystig Fritchie
58fa35a674 Remove escript-related proof-of-concept stuff from machi_util.erl 
I'd first thought that having that code there would be a kind of
useful reminder: please move me somewhere else.  However, there's
quite a bit there that's "cluster of clusters" stuff and not
appropriate for the current short-term work.
 2015-04-02 14:36:22 +09:00
Scott Lystig Fritchie
f8263c15cc Move FLU client 'verify checksums + local path' code from prototype/demo-day-hack 2015-04-02 12:38:12 +09:00
Scott Lystig Fritchie
76fcd4d931 Move FLU client 'verify checksums' code from prototype/demo-day-hack 2015-04-01 18:35:10 +09:00
Scott Lystig Fritchie
5c20ee6337 Fix client API for file list & checksum list 2015-04-01 18:02:16 +09:00
Scott Lystig Fritchie
d243ffca23 Single server client & server code (squashed) 2015-04-01 16:14:24 +09:00