Scott Lystig Fritchie
df8eea8c10
WIP: dict -> gb_trees, 1 of 2
2015-12-11 12:54:54 +09:00
Scott Lystig Fritchie
61eae1300f
WIP: finish basic 'run', add negative tests
2015-12-11 12:43:38 +09:00
Scott Lystig Fritchie
3ee3de1aaf
WIP: end of day
2015-12-10 23:44:27 +09:00
Scott Lystig Fritchie
6a5c590ad1
WIP: AST change {chain,...} thingie
2015-12-10 23:05:08 +09:00
Scott Lystig Fritchie
c37f23d97a
WIP: 'Run' AST thingie ha, take that, wheel!
2015-12-10 22:53:17 +09:00
Scott Lystig Fritchie
9cec53eea6
Yet another strawman AST
2015-12-10 19:18:25 +09:00
Shunichi Shinohara
eef00e4f8f
Add TODO comment for possible race condition
2015-12-10 15:58:17 +09:00
Scott Lystig Fritchie
9472bad37b
Clean up test errors
2015-12-10 15:57:35 +09:00
Scott Lystig Fritchie
61ef7739cd
Modify chain mostly works, better
2015-12-10 00:12:34 +09:00
Scott Lystig Fritchie
b0a9e65ca2
WIP: trying to shut down entire chain, but buggy, derp
2015-12-09 23:00:27 +09:00
Scott Lystig Fritchie
95e2df304e
WIP: minor cleanup
2015-12-09 22:25:43 +09:00
Scott Lystig Fritchie
7f25fcc8f8
Modify chain mostly works
2015-12-09 19:02:16 +09:00
Scott Lystig Fritchie
cd9bf9eeab
Modify chain mostly works, 2 TODOs remain
2015-12-09 18:17:26 +09:00
Shunichi Shinohara
2e2d282afc
Use outside of ephemeral port range to listen on
...
When there is TCP_WAIT connection whose local part has
port to be listened, listen (bind) will fail by eaddrinuse
_on Linux_ (won't on Mac OS X).
This commit also adds some logs and pattern matches.
Reference
- Ephemeral port - Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Ephemeral_port
"Many Linux kernels use the port range 32768 to 61000.[note 2]
FreeBSD has used the IANA port range since release 4.6. Previous
versions, including the Berkeley Software Distribution (BSD), use
ports 1024 to 5000 as ephemeral ports.[2]"
- Demostration of collision between already-closed ephemeral port
and listen port on Linux (Mac OS X allows)
https://gist.github.com/shino/36ae1e01608366d52236
2015-12-09 18:04:50 +09:00
Scott Lystig Fritchie
2871f8397c
WIP: modify chain still a bit broken
2015-12-09 17:19:02 +09:00
Scott Lystig Fritchie
65eec61f82
Basic stuff to add new flus via 'pending' dir
2015-12-09 14:48:46 +09:00
Scott Lystig Fritchie
7301c8308e
Clarify the initial docs, thanks @mrallen1!
2015-12-09 14:07:27 +09:00
Scott Lystig Fritchie
f23e500993
WIP comments
2015-12-09 11:32:05 +09:00
Shunichi Shinohara
ade4430d30
More cleaner clean up
2015-12-09 10:29:45 +09:00
Shunichi Shinohara
aa0a0413d1
Cosmetics of comments, specs, whitespaces and unit tests refactoring
2015-12-09 09:58:34 +09:00
Shunichi Shinohara
14765a7279
Change ranch callback module name
2015-12-09 09:58:34 +09:00
Shunichi Shinohara
a1f5a6ce62
Fix unit test cases around flu1 startup
2015-12-09 09:58:34 +09:00
Shunichi Shinohara
7614910f36
Initialize FLU package with ranch listener
2015-12-09 09:58:33 +09:00
Shunichi Shinohara
9579b1b8b2
Introduce ranch and add transport callback
2015-12-09 09:58:33 +09:00
Shunichi Shinohara
83b4466a85
Merge pull request #50 from basho/slf/doc-name-game2
...
Review & merge slf/doc-name-game2
2015-12-09 09:45:11 +09:00
Scott Lystig Fritchie
69280bfb4f
Fix typo/thinko: correct chain name @ bootstrap
2015-12-08 22:19:26 +09:00
Scott Lystig Fritchie
0fc7bc74b7
EDoc fixes
2015-12-08 22:05:11 +09:00
Scott Lystig Fritchie
8285899dba
Bootstrap chain @ app init: done, with an example.
...
For example:
% make clean
% make stage
And then configure 3 FLUs:
% echo '{p_srvr, a, machi_flu1_client, "localhost", 39000, []}.' > rel/machi/etc/flu-config/a
% echo '{p_srvr, b, machi_flu1_client, "localhost", 39001, []}.' > rel/machi/etc/flu-config/b
% echo '{p_srvr, c, machi_flu1_client, "localhost", 39002, []}.' > rel/machi/etc/flu-config/c
And then configure a chain to use 2 of those 3 FLUs:
% echo '{chain_def_v1,c1,ap_mode,[{p_srvr,a,machi_flu1_client,"localhost",39000,[]},{p_srvr,b,machi_flu1_client,"localhost",39001,[]}],[],[]}.' > rel/machi/etc/chain-config/c1
... then start Machi e.g.
% ./rel/machi/bin/machi console
... you should see the following console messages scroll by (including a :
=PROGRESS REPORT==== 8-Dec-2015::22:01:44 ===
supervisor: {local,machi_flu_sup}
started: [{pid,<0.145.0>},
{name,a},
{mfargs,
{machi_flu_psup,start_link,
[a,39000,"./data/flu/a",[]]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]
[... and also for the other two FLUs, including a bunch of progress
reports for processes that started underneath that sub-supervisor.]
22:01:44.446 [info] Running FLUs: [a,b,c]
22:01:44.446 [info] Running FLUs at epoch 0: [a,b,c]
22:01:44.532 [warning] The following FLUs are defined but are not also members of a defined chain: [c]
2015-12-08 21:57:29 +09:00
Scott Lystig Fritchie
37ac09a680
Rename src/machi_chain_bootstrap.erl -> src/machi_lifecycle_mgr.erl
2015-12-08 17:46:11 +09:00
Scott Lystig Fritchie
27e8a31307
Fix fitness timing problem with short-circuit +trigger_early_adjustment/2
2015-12-08 15:27:47 +09:00
Scott Lystig Fritchie
ef10ebed22
WIP: now trying to diagnose fitness server bug?
2015-12-08 14:50:16 +09:00
Scott Lystig Fritchie
1bc9033076
Yay, all tests pass!
2015-12-07 22:15:23 +09:00
Scott Lystig Fritchie
38e63e8181
Add & remove, mostly working (2 eunit tests broken)
2015-12-07 21:52:27 +09:00
UENISHI Kota
293eb4810f
Fix dialyzer error
2015-12-07 14:49:34 +09:00
Scott Lystig Fritchie
5aeaf872d9
WIP: machi_chain_manager1:set_chain_members() API change, all tests pass, yay
2015-12-07 14:41:56 +09:00
UENISHI Kota
89e80a8862
Fix GC not running
2015-12-07 12:07:46 +09:00
UENISHI Kota
07c2b97918
Change checksum_list API to return a t2b list
2015-12-07 10:55:45 +09:00
Scott Lystig Fritchie
3c880dc437
WIP: find 1st overlapping FLU in any #chain_def_v1{}
2015-12-04 17:47:18 +09:00
UENISHI Kota
befa776685
Fix several new bugs
2015-12-04 17:38:36 +09:00
Scott Lystig Fritchie
a7ffef6b8e
Add src/machi_chain_bootstrap.erl
2015-12-04 17:18:15 +09:00
UENISHI Kota
8528567954
Add eleveldb with sext to use it as metadata storage
...
First step is to use as checksum table. It will also used for file
names store and *ALL* other persistent metadata than files.
2015-12-04 16:38:57 +09:00
Scott Lystig Fritchie
cf0829b934
Add rc.d style config dir for FLU server startup
2015-12-04 16:37:05 +09:00
Scott Lystig Fritchie
35c48300a5
Fix Dialyzer complaints, derp!
2015-12-04 15:21:44 +09:00
Scott Lystig Fritchie
e9b1134cd9
Merge pull request #48 from basho/mra/merkle-cleanup
...
Add merkle library
2015-12-02 16:25:50 +09:00
Scott Lystig Fritchie
d44e9dd542
Fix plumbing for find_file()
2015-12-02 15:54:34 +09:00
Scott Lystig Fritchie
916ac754d7
WIP: still broken, almost passes suites=machi_cr_client_test tests=smoke_test_
2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
5477f3b6f8
WIP broken
2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
c3002d9852
Push TODO closer to actual TODO work site, standardize var spelling
2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
10a27ce7dd
All eunit tests now passing again
2015-12-02 15:35:38 +09:00
Scott Lystig Fritchie
014ba89e3a
Add client-side plumbing for high proto append chunk CoC
2015-12-02 15:35:38 +09:00
Mark Allen
a5a0369905
Add basic unit test
2015-11-20 21:22:41 -06:00
Mark Allen
4ce7a87d56
Remove merklet
2015-11-20 16:29:17 -06:00
UENISHI Kota
6786820401
Merge pull request #35 from basho/ku/making-file-proxy-spec
...
Add eqc trim tests to machi_file_proxy
2015-11-05 16:27:48 +09:00
Shunichi Shinohara
1b0711f151
Stop flu1 client under CR client
2015-11-05 11:46:22 +09:00
UENISHI Kota
d0e6417f5d
Remove unnecessary output
2015-11-04 16:15:33 +09:00
UENISHI Kota
3f6f9e2c6b
Address one offset+length issue
2015-11-04 16:08:43 +09:00
UENISHI Kota
c1e5426034
Address PR comments
2015-11-04 16:08:09 +09:00
UENISHI Kota
3b087c0388
Add eqc trim tests to machi_file_proxy
...
* Add description on high client APIs
* Add notes to rethink high client specification
2015-11-04 16:02:29 +09:00
Shunichi Shinohara
059f591d3f
Exclude FLUs which can not send to cr client as well as receive from
...
in terms of partition simulator
2015-11-02 17:57:16 +09:00
Mark Allen
3c5a9e6f53
Torture tests for merkle tree
...
1,000,000 entries - timings and size
2015-11-02 00:12:58 -06:00
Mark Allen
72a4fab49d
Add a naive diff function
2015-10-29 22:18:20 -05:00
Shunichi Shinohara
6fa2de28cd
Add self pid to debug print
2015-10-30 09:44:54 +09:00
Shunichi Shinohara
93b168415d
Change log level to debug for multiple files with certain seq num
2015-10-30 09:40:07 +09:00
Shunichi Shinohara
bf5768eb47
Make CR client partition-simulator-aware
2015-10-30 09:39:21 +09:00
Shunichi Shinohara
447c8c8d48
Add 2-tuple timeout setting to CR client
2015-10-30 09:39:21 +09:00
Scott Lystig Fritchie
611f33e81b
Change trigger_gc default -> 0
2015-10-29 15:05:15 +09:00
UENISHI Kota
028135d927
Update some comments for concise and sound description
2015-10-29 12:07:34 +09:00
Mark Allen
7086899941
Reorg merkle tree code into a library
...
Was a service previously. Now contains both merklet
and the naive implementations. Put construction
timing stuff into the test.
Tests are not truly meaningful yet.
2015-10-28 16:59:49 -05:00
UENISHI Kota
170b3cd797
Dialyzer fix
2015-10-28 12:48:50 +09:00
UENISHI Kota
f7358424e4
Trim command and GC prototype implementation
...
* maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
and the file is larger than `max_file_size`
* A file is deleted if all chunks except 1024 bytes header are trimmed
* If a file is going to be deleted, file_proxy notifies metadata_mgr
to remember the filename persistently, whose filename is
`known_files_<FluName>`
* Such trimmed filenames are stored in a machi_plist file per flu
* machi_file_proxy could not be started if the filename is in the
manager's list. Consequently, any write, read and trim operations
cannot happen against deleted file.
* After the file was trimmed, any read request to the file returns
`{error, trimmed}`
* Disclaimer: no tests written yet and machi_plist does not support
any recovery from partial writes.
* Add some thoughts as comments for repairing trims.
* State diagram of every byte is as follows:
```
state\action| write/append | read_chunk | trim_chunk
------------+----------------+------------------+---------------
unwritten | -> written | fail (+repair) | -> trimmed
written | noop or repair | return content | -> trimmed
trimmed | fail | fail | noop
```
2015-10-28 12:34:03 +09:00
Mark Allen
5e571f6009
Switch to merklet
...
Still a WIP
2015-10-27 16:33:18 -05:00
Mark Allen
b710517c64
Fixes after testing
2015-10-27 11:57:38 -05:00
Mark Allen
48fabdcd5f
WIP
...
Almost certainly broken
2015-10-27 11:57:38 -05:00
Scott Lystig Fritchie
61f02dfc9f
Merge pull request #30 from basho/ss-fix-opt-timeout-arg-mismatch
...
Fix missing "options" arg for a few calls
2015-10-27 14:36:52 +09:00
Scott Lystig Fritchie
5ec35773e3
Merge pull request #28 from basho/ku/csum-table-anybytes
...
Change machi_csum_table to support arbitrary bytes writes and trims
2015-10-27 14:21:09 +09:00
Scott Lystig Fritchie
d2b1c7512a
Merge branch 'ku/config-system' into tmp
2015-10-27 14:14:40 +09:00
Scott Lystig Fritchie
b500d5f449
config hell
2015-10-27 14:07:45 +09:00
Scott Lystig Fritchie
dd17b1de0a
Remove compiler warnings
2015-10-27 14:07:07 +09:00
UENISHI Kota
7377624579
Dialyzer cleanup
2015-10-27 13:44:12 +09:00
UENISHI Kota
6e3347f727
Support log replay
2015-10-27 13:43:45 +09:00
UENISHI Kota
8a61055f55
Support arbitrary bytes write by using find_(left|right)neighbor/2
2015-10-27 13:43:45 +09:00
UENISHI Kota
d59c1fae31
Fix dialyzer issue
2015-10-27 13:27:20 +09:00
Shunichi Shinohara
39ac71048b
Fix missing "options" arg for a few calls
2015-10-27 13:03:56 +09:00
Scott Lystig Fritchie
bbbd9748f3
Fix compiler & dialyzer warnings
2015-10-27 12:45:48 +09:00
UENISHI Kota
b2eb3e089c
Cleanup MACROs and changed default value
...
* machi_file_proxy now uses application environment
value `max_file_size` via machi_config
* changed name from MAX_FILE_SIZE to DEFAULT_MAX_FILE_SIZE
2015-10-27 11:17:59 +09:00
UENISHI Kota
5913531e32
Introduce machi_config.erl
2015-10-27 11:00:05 +09:00
UENISHI Kota
60364fe0ca
Fix typo
2015-10-26 10:10:08 +09:00
UENISHI Kota
0e4ae818af
Clarify checksum tags, define macros to avoid typos
2015-10-23 19:32:22 +09:00
UENISHI Kota
3d6d4d8be3
Do the slicing in flu server rather than in CR client
2015-10-23 18:49:49 +09:00
UENISHI Kota
c5661571e3
Regenerate checksum when chunks are to be sliced
2015-10-23 17:33:08 +09:00
UENISHI Kota
5fd225bcdb
Merge pull request #25 from basho/ku/trim-pb-protocol-2
...
Update read_chunk() PB protocol to return trimmed chunks
2015-10-23 17:10:21 +09:00
UENISHI Kota
0f688d6279
Update read_chunk() PB protocol to return trimmed chunks
2015-10-22 23:11:43 +09:00
Scott Lystig Fritchie
49b4b1c304
Silence remaining warnings
2015-10-22 15:37:09 +09:00
Scott Lystig Fritchie
a0588cbaed
Avoid warnings
2015-10-22 12:59:40 +09:00
Scott Lystig Fritchie
3bb5ffa50e
Remove src/machi_sequencer.erl
2015-10-22 12:56:24 +09:00
Scott Lystig Fritchie
b8c5d21876
cp_mode repair warning fix
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
9c31139b62
Whitespace
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
b3b24b1178
Type corrections & remove dead code
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
5122ee00a9
Bugfixes
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
0c4c42cc52
Bugfixes
2015-10-21 18:37:30 +09:00