Shunichi Shinohara
6fa2de28cd
Add self pid to debug print
2015-10-30 09:44:54 +09:00
Shunichi Shinohara
93b168415d
Change log level to debug for multiple files with certain seq num
2015-10-30 09:40:07 +09:00
Shunichi Shinohara
bf5768eb47
Make CR client partition-simulator-aware
2015-10-30 09:39:21 +09:00
Shunichi Shinohara
447c8c8d48
Add 2-tuple timeout setting to CR client
2015-10-30 09:39:21 +09:00
Scott Lystig Fritchie
611f33e81b
Change trigger_gc default -> 0
2015-10-29 15:05:15 +09:00
UENISHI Kota
028135d927
Update some comments for concise and sound description
2015-10-29 12:07:34 +09:00
Mark Allen
7086899941
Reorg merkle tree code into a library
...
Was a service previously. Now contains both merklet
and the naive implementations. Put construction
timing stuff into the test.
Tests are not truly meaningful yet.
2015-10-28 16:59:49 -05:00
UENISHI Kota
170b3cd797
Dialyzer fix
2015-10-28 12:48:50 +09:00
UENISHI Kota
f7358424e4
Trim command and GC prototype implementation
...
* maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
and the file is larger than `max_file_size`
* A file is deleted if all chunks except 1024 bytes header are trimmed
* If a file is going to be deleted, file_proxy notifies metadata_mgr
to remember the filename persistently, whose filename is
`known_files_<FluName>`
* Such trimmed filenames are stored in a machi_plist file per flu
* machi_file_proxy could not be started if the filename is in the
manager's list. Consequently, any write, read and trim operations
cannot happen against deleted file.
* After the file was trimmed, any read request to the file returns
`{error, trimmed}`
* Disclaimer: no tests written yet and machi_plist does not support
any recovery from partial writes.
* Add some thoughts as comments for repairing trims.
* State diagram of every byte is as follows:
```
state\action| write/append | read_chunk | trim_chunk
------------+----------------+------------------+---------------
unwritten | -> written | fail (+repair) | -> trimmed
written | noop or repair | return content | -> trimmed
trimmed | fail | fail | noop
```
2015-10-28 12:34:03 +09:00
Mark Allen
5e571f6009
Switch to merklet
...
Still a WIP
2015-10-27 16:33:18 -05:00
Mark Allen
b710517c64
Fixes after testing
2015-10-27 11:57:38 -05:00
Mark Allen
48fabdcd5f
WIP
...
Almost certainly broken
2015-10-27 11:57:38 -05:00
Scott Lystig Fritchie
61f02dfc9f
Merge pull request #30 from basho/ss-fix-opt-timeout-arg-mismatch
...
Fix missing "options" arg for a few calls
2015-10-27 14:36:52 +09:00
Scott Lystig Fritchie
5ec35773e3
Merge pull request #28 from basho/ku/csum-table-anybytes
...
Change machi_csum_table to support arbitrary bytes writes and trims
2015-10-27 14:21:09 +09:00
Scott Lystig Fritchie
d2b1c7512a
Merge branch 'ku/config-system' into tmp
2015-10-27 14:14:40 +09:00
Scott Lystig Fritchie
b500d5f449
config hell
2015-10-27 14:07:45 +09:00
Scott Lystig Fritchie
dd17b1de0a
Remove compiler warnings
2015-10-27 14:07:07 +09:00
UENISHI Kota
7377624579
Dialyzer cleanup
2015-10-27 13:44:12 +09:00
UENISHI Kota
6e3347f727
Support log replay
2015-10-27 13:43:45 +09:00
UENISHI Kota
8a61055f55
Support arbitrary bytes write by using find_(left|right)neighbor/2
2015-10-27 13:43:45 +09:00
UENISHI Kota
d59c1fae31
Fix dialyzer issue
2015-10-27 13:27:20 +09:00
Shunichi Shinohara
39ac71048b
Fix missing "options" arg for a few calls
2015-10-27 13:03:56 +09:00
Scott Lystig Fritchie
bbbd9748f3
Fix compiler & dialyzer warnings
2015-10-27 12:45:48 +09:00
UENISHI Kota
b2eb3e089c
Cleanup MACROs and changed default value
...
* machi_file_proxy now uses application environment
value `max_file_size` via machi_config
* changed name from MAX_FILE_SIZE to DEFAULT_MAX_FILE_SIZE
2015-10-27 11:17:59 +09:00
UENISHI Kota
5913531e32
Introduce machi_config.erl
2015-10-27 11:00:05 +09:00
UENISHI Kota
60364fe0ca
Fix typo
2015-10-26 10:10:08 +09:00
UENISHI Kota
0e4ae818af
Clarify checksum tags, define macros to avoid typos
2015-10-23 19:32:22 +09:00
UENISHI Kota
3d6d4d8be3
Do the slicing in flu server rather than in CR client
2015-10-23 18:49:49 +09:00
UENISHI Kota
c5661571e3
Regenerate checksum when chunks are to be sliced
2015-10-23 17:33:08 +09:00
UENISHI Kota
5fd225bcdb
Merge pull request #25 from basho/ku/trim-pb-protocol-2
...
Update read_chunk() PB protocol to return trimmed chunks
2015-10-23 17:10:21 +09:00
UENISHI Kota
0f688d6279
Update read_chunk() PB protocol to return trimmed chunks
2015-10-22 23:11:43 +09:00
Scott Lystig Fritchie
49b4b1c304
Silence remaining warnings
2015-10-22 15:37:09 +09:00
Scott Lystig Fritchie
a0588cbaed
Avoid warnings
2015-10-22 12:59:40 +09:00
Scott Lystig Fritchie
3bb5ffa50e
Remove src/machi_sequencer.erl
2015-10-22 12:56:24 +09:00
Scott Lystig Fritchie
b8c5d21876
cp_mode repair warning fix
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
9c31139b62
Whitespace
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
b3b24b1178
Type corrections & remove dead code
2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
5122ee00a9
Bugfixes
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
0c4c42cc52
Bugfixes
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
d2ac5b0583
Bugfix: arg type to machi_util:parse_filename()
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
028ddc79ff
Data type cleanups, other
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
595f9a463e
Unexported funcs
2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
177aca0a68
Merge pull request #22 from basho/ss-flu1-init-sync
...
Make flu1 initialization synchronous
2015-10-21 18:36:12 +09:00
Shunichi Shinohara
478107915b
Make flu1 initialization synchronous
2015-10-21 16:16:03 +09:00
Scott Lystig Fritchie
30d7e592a3
Merge pull request #20 from basho/ku/read-all-chunks
...
Allow reading multiple chunks at once
2015-10-21 15:28:10 +09:00
Scott Lystig Fritchie
1c8e436a64
Fix race #3
2015-10-21 15:01:11 +09:00
UENISHI Kota
a43397a7b8
Update to review comments
2015-10-21 10:58:00 +09:00
UENISHI Kota
ebb9bc3f5a
Allow reading multiple chunks at once
...
* When repairing multiple chunks at once and any of its repair
failed, the whole read request and repair work will fail
* Rename read_repair3 and read_repair4 to do_repair_chunks and
do_repair chunk in machi_file_proxy
* This pull request changes return semantics of read_chunk(), that
returns any chunk included in requested range
* First and last chunk may be cut to fit the requested range
* In machi_file_proxy, unwritten_bytes are removed and replaced by
machi_csum_table
2015-10-20 17:59:09 +09:00
Scott Lystig Fritchie
6f9814ffb4
Merge ss/deps-for-debugging (with rebar.config conflict fix)
2015-10-19 16:41:03 +09:00
UENISHI Kota
3e975f53b8
Allow read_chunk() to return partial chunks
...
This is simply a change of read_chunk() protocol, where a response of
read_chunk() becomes list of written bytes along with checksum. All
related code including repair is changed as such. This is to pass all
tests and not actually supporting partial chunks.
2015-10-19 15:37:17 +09:00
Shunichi Shinohara
208c02853f
Add cluster_info to deps and small callback module
...
For debuging from shell, some functions in machi_cinfo are exported:
- public_projection/1
- private_projection/1
- fitness/1
- chain_manager/1
- flu1/1
2015-10-19 15:36:05 +09:00
UENISHI Kota
cb67764273
Merge pull request #12 from basho/slf/packaging1
...
Slf/packaging1
2015-10-16 17:56:27 +09:00
Scott Lystig Fritchie
00ac0f4cd3
Reduce compiler warnings and verbose output that clutters eunit test output
2015-10-16 17:41:01 +09:00
Scott Lystig Fritchie
299016cafb
FLU startup via app.config
2015-10-16 16:28:46 +09:00
UENISHI Kota
6f790527f5
Follow with missing tests and related fix
2015-10-16 10:10:05 +09:00
UENISHI Kota
e45469b5ce
Move checksum file related code to machi_csum_table
2015-10-15 11:28:40 +09:00
Mark Allen
baeffbab0b
Merge pull request #6 from basho/mra/write-once-clean
...
Integrate write once invariant into current FLU implementation
2015-10-14 10:15:57 -05:00
Scott Lystig Fritchie
e344ee42ff
Remove stale TODO comment about write-once enforcement
2015-10-14 16:56:51 +09:00
Scott Lystig Fritchie
d6a3180ecd
Use pattern matching instead of length() BIF
2015-10-14 16:52:03 +09:00
UENISHI Kota
07ceff095a
Fix gen_server style return value
2015-10-14 16:22:11 +09:00
Scott Lystig Fritchie
8eb9cc9700
Fix "HEY, machi_pb_translate:852 got {error,bad_csum}" errors
...
s/bad_csum/bad_checksum/ as needed in in machi_file_proxy.erl
2015-10-14 14:26:46 +09:00
Scott Lystig Fritchie
ed112bfb52
Argument fix for read_chunk() when write_chunk() says 'written'
2015-10-14 14:16:51 +09:00
Scott Lystig Fritchie
6dbf52db6f
Remove some debugging verbosity
2015-10-14 12:50:10 +09:00
UENISHI Kota
1b612bd969
Fix typo in comment
2015-10-14 12:40:56 +09:00
Mark Allen
fe71b72494
Add filename parse and validation functions
2015-10-13 21:12:14 -05:00
Mark Allen
f8707c61c0
Choose new filename when epoch changes
...
The filename manager needs to choose a new file name
for a prefix when the epoch number changes. This helps
ensure safety of file merges across the cluster.
(Prevents conflicts across divergent cluster members.)
2015-10-13 21:09:31 -05:00
Mark Allen
161e6cd9f9
Pass epoch id to append operations
...
Needed to handle a filename change when epoch changes.
2015-10-13 21:08:48 -05:00
Mark Allen
85e1e5a26d
Handle {error, bad_arg} on read
2015-10-13 21:08:24 -05:00
UENISHI Kota
e113f6ffdd
Reach the trim stub to CR client
2015-10-13 17:25:59 +09:00
UENISHI Kota
dfe953b7d8
Add surface of trim to scrub
2015-10-13 17:14:44 +09:00
Scott Lystig Fritchie
777909b0f5
TODO MARK todo comment and bugfix for machi_cr_client_test
2015-10-12 15:30:37 +09:00
Mark Allen
289b2bcc7c
Debug WIP
2015-10-11 23:04:29 -05:00
Mark Allen
c1b9038447
The return value of ets is generally 'true'
2015-10-08 15:47:11 -05:00
Mark Allen
aca3759e45
Bug fixes found during testing runs
2015-10-08 15:46:40 -05:00
Mark Allen
1ecbb5cffe
Fixed order of start_link parameters
2015-10-08 15:45:04 -05:00
Mark Allen
303aad97e9
Use {error, bad_checksum} directly
...
We previously copied {error, bad_csum} as it was used in the main
FLU code. The protobufs stuff expects the full atom bad_checksum
though.
2015-10-08 15:43:54 -05:00
Scott Lystig Fritchie
952d2fa508
Change flag_checksum -> flag_no_checksum for consistency
2015-10-08 20:41:59 +09:00
Mark Allen
679046600f
Merge remote-tracking branch 'origin/bug/from-bp-request-error' into mra/write-once-clean
2015-10-07 23:02:03 -05:00
Scott Lystig Fritchie
796937fe75
Add LL generic error PB response decoding
2015-10-08 12:33:55 +09:00
Scott Lystig Fritchie
0054445f13
Delete spammy message from fitness servers every 5 seconds
2015-10-07 18:52:24 +09:00
Mark Allen
d627f238bf
Cache generated names until disk files are written
2015-10-06 22:44:31 -05:00
Mark Allen
f83b0973f2
Have to call filename mgr with FluName
2015-10-06 22:43:19 -05:00
Mark Allen
7a6999465a
Make sure we use '^' as filename separators
2015-10-06 22:02:31 -05:00
Mark Allen
2d0c03ef35
Integration with current FLU implementation
2015-10-05 22:18:29 -05:00
Mark Allen
36c11e7d08
Add a metadata manager supervisor
2015-10-05 16:37:53 -05:00
Mark Allen
d3fe7ee181
Pull write-once files over to clean branch
...
I am treating the original write-once branch as a prototype
which I am now throwing away. I had too much work interleved
in there, so I felt like the best thing to do would be to cut
a new clean branch and pull the files over and start over
against a recent-ish master.
We will have to refactor the other things in FLU in a more
piecemeal fashion.
2015-10-02 16:29:09 -05:00
Scott Lystig Fritchie
6d5b61f747
Tweaks to sleep_ranked_order() call in C200
2015-09-21 21:47:25 +09:00
Scott Lystig Fritchie
5eecb2b935
Change to P_current_calc epoch @ C100
2015-09-21 21:44:03 +09:00
Scott Lystig Fritchie
340af05f0f
WIP: server-side of CP mode repairing-as-witness
2015-09-21 21:44:03 +09:00
Scott Lystig Fritchie
d9b9397e75
Avoid some projection churn in C100's sanity check
2015-09-21 21:44:03 +09:00
Scott Lystig Fritchie
5010d03677
Call manage_last_down_list() at C220 and C310
2015-09-21 15:36:54 +09:00
Scott Lystig Fritchie
69a304102e
Write public proj in all_members order only
2015-09-21 15:09:16 +09:00
Scott Lystig Fritchie
6b4ed1c061
Verbose debugging cruft
2015-09-19 14:25:07 +09:00
Scott Lystig Fritchie
72bfa163ba
Small test bugfixes & verbose/debugging cruft
2015-09-19 14:16:54 +09:00
Scott Lystig Fritchie
d695f30e4f
Avoid using host/port combo for machi_fitness (ab)use of machi_projection
2015-09-17 16:43:08 +09:00
Scott Lystig Fritchie
09ae2db0ba
Bugfix: double-check local private projection write with a read
2015-09-16 16:31:10 +09:00
Scott Lystig Fritchie
79b1d156c4
Add backlog option to gen_tcp:listen
2015-09-16 13:52:36 +09:00
Scott Lystig Fritchie
778bd015ee
Bugfix: pattern matching error in C110
2015-09-16 12:41:53 +09:00
Scott Lystig Fritchie
d3b116bd9e
Bugfix: CP mode: ignore P_latest if it has UPI or down server in my down list
2015-09-15 17:55:18 +09:00
Scott Lystig Fritchie
75c94420e0
Add test_ets_table to give programmatic slowdown
2015-09-14 22:52:41 +09:00