f().
exit(self(), bye).
{ok, T} = machi_merkle_tree_mgr:start_link(a, "/tmp/a", []).
Leaves = [{{Pos*65,65}, begin {A,B,C} = now(), crypto:hash(sha, <<A:32, B:32, C:32>>) end} || Pos <- lists:seq(1,10*1000)]. length(Leaves).
timer:tc(fun() -> [machi_merkle_tree_mgr:update(a, "yofile-3", Off, Sz, CSum) || {{Off,Sz},CSum} <- Leaves] end).
timer:tc(fun() -> machi_merkle_tree_mgr:fetch(a, "yofile-3") end).
All of the update() calls are quick, but they are gen_server:cast()
calls! So, when we send the fetch(), we see that the gen_server proc
doesn't get the {fetch,...} tuple for 7 seconds. All of the update
stuff is taking lots of time. The tree building itself is a tiny
fraction of the total time.
7> timer:tc(fun() -> machi_merkle_tree_mgr:fetch(a, "yofile-3") end).
build_tree LINE 136 at {1446,95570,34679}
build_tree LINE 152 at {1446,95577,180438}
build_tree LINE 306 at {1446,95577,180770}
build_tree LINE 308 at {1446,95577,181340}
build_tree LINE 311 at {1446,95577,185524}
build_tree LINE 315 at {1446,95577,185572}
build_tree LINE 319 at {1446,95577,185589}
build_tree LINE 322 at {1446,95577,185608}
Root: <<95,199,42,232,84,255,228,36,2,174,24,5,20,56,69,109,109,26,92,220>>
{7151003,
{ok,[<<95,199,42,232,84,255,228,36,2,174,24,5,20,56,69,
[...]
Example CLI session:
f(Leaves).
Leaves = [{{Pos*65,65}, begin {A,B,C} = now(), crypto:hash(sha, <<A:32, B:32, C:32>>) end} || Pos <- lists:seq(1,1*1000*1000)].
timer:tc(fun() -> catch machi_merkle_tree_mgr:build_tree(x, {mt, "/tmp/foo", true, undefined, [], [], [], Leaves}) end).
On my MPB, I see about 550000 - 650000 msec, depending on the run. I've
made no attempt yet to be more rigorous about making measurements.
A result looks like this. If we ignore the timer:tc() tuple wrapper and
time report, then the output list is [Root, Lvl3, Lvl2, Lvl1].
Root: <<250,83,19,148,149,195,105,253,231,116,144,49,227,168,224,51,76,63,162,
70>>
{551957,
[<<250,83,19,148,149,195,105,253,231,116,144,49,227,168,
224,51,76,63,162,70>>,
[<<102,159,171,226,30,76,65,38,81,168,229,143,240,112,
208,13,169,216,57,35>>],
[<<67,189,3,58,14,201,74,206,240,202,163,83,182,232,130,
14,107,29,84,181>>],
[<<210,57,128,48,28,28,87,185,240,55,129,158,119,77,172,0,
183,166,205,157>>,
<<78,154,155,117,1,220,48,116,74,66,172,146,29,39,7,7,247,
188,142,72>>,
<<4,50,223,192,38,141,54,249,168,79,213,90,94,32,91,117,
36,186,41,139>>,
<<20,181,105,237,50,97,98,91,178,26,127,20,134,174,146,96,
133,208,245,...>>,
<<227,25,30,162,182,226,172,29,132,240,125,248,65,109,82,
6,116,102,...>>,
<<229,221,31,166,106,212,120,146,96,229,224,85,237,206,
160,38,137,...>>,
<<124,195,225,167,238,245,75,161,71,53,43,154,103,188,
180,171,...>>]]}
* When repairing multiple chunks at once and any of its repair
failed, the whole read request and repair work will fail
* Rename read_repair3 and read_repair4 to do_repair_chunks and
do_repair chunk in machi_file_proxy
* This pull request changes return semantics of read_chunk(), that
returns any chunk included in requested range
* First and last chunk may be cut to fit the requested range
* In machi_file_proxy, unwritten_bytes are removed and replaced by
machi_csum_table
This is simply a change of read_chunk() protocol, where a response of
read_chunk() becomes list of written bytes along with checksum. All
related code including repair is changed as such. This is to pass all
tests and not actually supporting partial chunks.
For debuging from shell, some functions in machi_cinfo are exported:
- public_projection/1
- private_projection/1
- fitness/1
- chain_manager/1
- flu1/1
