Commit graph

1090 commits

Author SHA1 Message Date
UENISHI Kota
e11cdfe95c Add stop and trim command to eqc_statem test on file_proxy 2015-11-18 14:11:25 +09:00
Mark Allen
8d3f631d84 Oops. Accidentally left this out. 2015-11-17 22:07:28 -06:00
UENISHI Kota
84058f8c9c Merge pull request #44 from basho/ss/repair-eqc-pleak-fix
Fix process leak of repair eqc
2015-11-17 19:16:24 +09:00
Shunichi Shinohara
ad419ada50 Refactoring, cosmetics, comments 2015-11-17 12:58:54 +09:00
Shunichi Shinohara
049311614f Fix process leak of CR clients and FLU1 proxy clients 2015-11-17 12:58:50 +09:00
Scott Lystig Fritchie
3a35fe38c8 Merge branch 'slf/doc-cleanup2' ... in the middle of things 2015-11-06 07:22:38 -08:00
Scott Lystig Fritchie
73890171ba Format PDF version of high-level-chain-mgr doc 2015-11-06 07:21:44 -08:00
Shunichi Shinohara
919a408e17 Ubuntu /bin/sh is dash then something wrong happens sometimes
It seems dash does not understand {a,b,...} file pattern ...

% echo 'cp /home/shino/local/erlang/17.5.6_basho_hipe/lib/erlang/lib/stdlib-*/src/{lists,dict}.erl ./.ebin.native' | sh -x                                               + cp /home/shino/local/erlang/17.5.6_basho_hipe/lib/erlang/lib/stdlib-*/src/{lists,dict}.erl ./.ebin.native
cp: cannot stat ‘/home/shino/local/erlang/17.5.6_basho_hipe/lib/erlang/lib/stdlib-*/src/{lists,dict}.erl’: No such file or directory
2015-11-06 12:35:02 +09:00
UENISHI Kota
6786820401 Merge pull request #35 from basho/ku/making-file-proxy-spec
Add eqc trim tests to machi_file_proxy
2015-11-05 16:27:48 +09:00
UENISHI Kota
81fae32539 Remove unused test function 2015-11-05 16:19:46 +09:00
UENISHI Kota
2e6e6dd9e8 Merge pull request #40 from basho/ss/missing-test-cleanups
Add missing test cleanups
2015-11-05 16:16:40 +09:00
UENISHI Kota
ce41f9005e Fix machi_file_proxy_eqc:write_post to proper assertion 2015-11-05 14:48:35 +09:00
Shunichi Shinohara
922baaf433 Make rebar unit output verbose 2015-11-05 11:55:47 +09:00
Shunichi Shinohara
9e4dc83f2a Add missing cleanup tasks, suppress some not-so-useful logs 2015-11-05 11:47:47 +09:00
Shunichi Shinohara
1b0711f151 Stop flu1 client under CR client 2015-11-05 11:46:22 +09:00
Shunichi Shinohara
39a937db9b Merge pull request #38 from basho/ku/otp-18
Support OTP 18.1
2015-11-05 10:49:56 +09:00
UENISHI Kota
f56037240e Plan trim commands in eqc tests 2015-11-04 16:32:53 +09:00
UENISHI Kota
d0e6417f5d Remove unnecessary output 2015-11-04 16:15:33 +09:00
UENISHI Kota
3f6f9e2c6b Address one offset+length issue 2015-11-04 16:08:43 +09:00
UENISHI Kota
c1e5426034 Address PR comments 2015-11-04 16:08:09 +09:00
UENISHI Kota
3b087c0388 Add eqc trim tests to machi_file_proxy
* Add description on high client APIs
* Add notes to rethink high client specification
2015-11-04 16:02:29 +09:00
UENISHI Kota
62c8dacc65 Merge pull request #33 from basho/ss-repair-with-partition-simulator
Add test for append and repair with partition simulator
2015-11-04 14:44:04 +09:00
UENISHI Kota
b956f9e1f0 Fix dialyzer issue 2015-11-04 11:43:00 +09:00
Scott Lystig Fritchie
850a8786b6 Merge branch 'slf/doc-converge-demo'
Docs only.
2015-11-03 00:37:53 +09:00
Scott Lystig Fritchie
557525af05 Clarify chain length 2015-11-03 00:34:51 +09:00
Scott Lystig Fritchie
30000d6602 Add doc/machi_chain_manager1_converge_demo.md 2015-11-03 00:27:09 +09:00
Shunichi Shinohara
059f591d3f Exclude FLUs which can not send to cr client as well as receive from
in terms of partition simulator
2015-11-02 17:57:16 +09:00
UENISHI Kota
3122f2cf54 Support OTP 18.1 2015-11-02 17:43:38 +09:00
Mark Allen
3c5a9e6f53 Torture tests for merkle tree
1,000,000 entries - timings and size
2015-11-02 00:12:58 -06:00
Mark Allen
72a4fab49d Add a naive diff function 2015-10-29 22:18:20 -05:00
Shunichi Shinohara
b5005c3526 Add EQC test case for AP mode repair w/ part. sim. 2015-10-30 10:25:51 +09:00
Shunichi Shinohara
6fa2de28cd Add self pid to debug print 2015-10-30 09:44:54 +09:00
Shunichi Shinohara
93b168415d Change log level to debug for multiple files with certain seq num 2015-10-30 09:40:07 +09:00
Shunichi Shinohara
bf5768eb47 Make CR client partition-simulator-aware 2015-10-30 09:39:21 +09:00
Shunichi Shinohara
447c8c8d48 Add 2-tuple timeout setting to CR client 2015-10-30 09:39:21 +09:00
Scott Lystig Fritchie
b859e23a37 Some long-overdue minor editing, prior to working on issue #4 2015-10-29 21:21:05 +09:00
Scott Lystig Fritchie
44497d5af8 Merge pull request #32 from basho/ku/trim-and-gc
Trim command and GC prototype implementation
2015-10-29 15:07:10 +09:00
Scott Lystig Fritchie
611f33e81b Change trigger_gc default -> 0 2015-10-29 15:05:15 +09:00
UENISHI Kota
028135d927 Update some comments for concise and sound description 2015-10-29 12:07:34 +09:00
Mark Allen
7086899941 Reorg merkle tree code into a library
Was a service previously. Now contains both merklet
and the naive implementations.  Put construction
timing stuff into the test.

Tests are not truly meaningful yet.
2015-10-28 16:59:49 -05:00
UENISHI Kota
170b3cd797 Dialyzer fix 2015-10-28 12:48:50 +09:00
UENISHI Kota
f7358424e4 Trim command and GC prototype implementation
* maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
  and the file is larger than `max_file_size`
* A file is deleted if all chunks except 1024 bytes header are trimmed
* If a file is going to be deleted, file_proxy notifies metadata_mgr
  to remember the filename persistently, whose filename is
  `known_files_<FluName>`
* Such trimmed filenames are stored in a machi_plist file per flu
* machi_file_proxy could not be started if the filename is in the
  manager's list. Consequently, any write, read and trim operations
  cannot happen against deleted file.
* After the file was trimmed, any read request to the file returns
  `{error, trimmed}`
* Disclaimer: no tests written yet and machi_plist does not support
  any recovery from partial writes.
* Add some thoughts as comments for repairing trims.

* State diagram of every byte is as follows:

```
state\action| write/append   | read_chunk       | trim_chunk
------------+----------------+------------------+---------------
 unwritten  |  -> written    | fail (+repair)   | -> trimmed
 written    | noop or repair | return content   | -> trimmed
 trimmed    |  fail          | fail             | noop
```
2015-10-28 12:34:03 +09:00
Mark Allen
5e571f6009 Switch to merklet
Still a WIP
2015-10-27 16:33:18 -05:00
Mark Allen
7f561f34e0 Ignore vim cruft too 2015-10-27 16:33:07 -05:00
Mark Allen
77096c5f82 Add merklet as a dependency 2015-10-27 11:57:38 -05:00
Mark Allen
b710517c64 Fixes after testing 2015-10-27 11:57:38 -05:00
Mark Allen
1b8401e7de Initial smoke test 2015-10-27 11:57:38 -05:00
Mark Allen
48fabdcd5f WIP
Almost certainly broken
2015-10-27 11:57:38 -05:00
Scott Lystig Fritchie
61f02dfc9f Merge pull request #30 from basho/ss-fix-opt-timeout-arg-mismatch
Fix missing "options" arg for a few calls
2015-10-27 14:36:52 +09:00
Scott Lystig Fritchie
5ec35773e3 Merge pull request #28 from basho/ku/csum-table-anybytes
Change machi_csum_table to support arbitrary bytes writes and trims
2015-10-27 14:21:09 +09:00