[WIP] Unify LevelDB usage to single instance

kuenishi commented

2016-02-09 04:42:02 +00:00

(Migrated from github.com)

Perfile LevelDB instance usage are changed to use single instance
per FLU server.
machi_csum_file reference is managed with machi_flu_filename_mgr
as an aim to manage filenames with leveldb
Not only chunk checksums, but the list of trimmed files are also
stored in LevelDB.
Remove 1024 bytes file header; instead put any metadata into
LevelDB if needed.
LevelDB db_ref() lifecycle is same as that of machi_metadata_mgr
machi_file_proxy just uses it as it's passed at process startup
There are several optimization space still left as it is

This is still WIP but it might be hard to continue the work for me. Currently this has 14 tests failing, mostly minor issues like mismatch of lists given to a function that only accepts binaries.

- Perfile LevelDB instance usage are changed to use single instance per FLU server. - machi_csum_file reference is managed with machi_flu_filename_mgr as an aim to manage filenames with leveldb - Not only chunk checksums, but the list of trimmed files are also stored in LevelDB. - Remove 1024 bytes file header; instead put any metadata into LevelDB if needed. - LevelDB `db_ref()` lifecycle is same as that of `machi_metadata_mgr` - `machi_file_proxy` just uses it as it's passed at process startup - There are several optimization space still left as it is This is still WIP but it might be hard to continue the work for me. Currently this has 14 tests failing, mostly minor issues like mismatch of lists given to a function that only accepts binaries.

kuenishi commented

2016-02-09 04:42:27 +00:00

(Migrated from github.com)

Also, I don't have permissions to push to machi original repo, so I'm opening a PR from my fork.

slfritchie commented

2016-02-16 05:37:24 +00:00

(Migrated from github.com)

Heh, I see fewer than 14 test failuers ... I see 6. Or 7, depending on the run.

*failed*
in function machi_proxy_flu1_client_test:api_smoke_test/0 (test/machi_proxy_flu1_client_test.erl, line 72)
**error:{badmatch,{error,partition}}

  machi_flu1_test: flu_smoke_test...*failed*
in function machi_flu1_test:flu_smoke_test/0 (test/machi_flu1_test.erl, line 124)
**error:{badmatch,{ok,{[{"prefix!^^00000000^e31aae63-9f81-46a0-9baa-60c66b72d7b0^0",0,
                 <<"yo!">>,
                 <<2,139,98,163,106,247,61,129,61,...>>}],
               []}}}

  machi_file_proxy_eqc:41: eqc_test_...*failed*
in function eqc:quickcheck/1 (../src/eqc.erl, line 1270)
in call from machi_file_proxy_eqc:'-eqc_test_/0-fun-1-'/1 (test/machi_file_proxy_eqc.erl, line 41)
**exit:{badarg,[{eleveldb,async_get,
                   [#Ref<0.0.0.40751>,<<>>,
                    <<16,0,0,0,2,12,179,90,109,150,...>>,
                    []],
                   []},
         {eleveldb,get,3,[{file,"src/eleveldb.erl"},{line,152}]},
         {machi_csum_table,is_file_trimmed,2,
                           [{file,"src/machi_csum_table.erl"},{line,158}]},
         {machi_file_proxy,init,1,
                           [{file,"src/machi_file_proxy.erl"},{line,224}]},
         {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]},
         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}

.*failed*
in function eqc:quickcheck/1 (../src/eqc.erl, line 1270)
in call from machi_ap_repair_eqc:'-prop_repair_test_/0-fun-1-'/3 (test/machi_ap_repair_eqc.erl, line 93)
**exit:{timeout,
    {gen_server,call,
        [<0.10322.0>,
         {req,
             {append_chunk_extra,
                 {1130,<<225,248,169,11,186,142,...>>},
                 [],0,<<"pre">>,
                 <<2,47,38,147,...>>,
                 0}},
         2000]}}

  machi_ap_repair_eqc:101: prop_repair_par_test_...*failed*
in function eqc:quickcheck/1 (../src/eqc.erl, line 1270)
in call from machi_ap_repair_eqc:'-prop_repair_par_test_/0-fun-1-'/3 (test/machi_ap_repair_eqc.erl, line 103)
**exit:{timeout,
    {gen_server,call,
        [<0.11008.0>,
         {req,
             {append_chunk_extra,
                 {1115,<<168,193,7,104,172,160,...>>},
                 [],0,<<"pre">>,
                 <<169,152,13,55,...>>,
                 0}},
         2000]}}


  [done in 18.022 s]
module 'machi_admin_util_test'
  machi_admin_util_test: verify_file_checksums_test_...*failed*
in function machi_util:wait_for_death/2 (src/machi_util.erl, line 357)
in call from machi_test_util:start_flu_packages/1 (test/machi_test_util.erl, line 56)
in call from machi_admin_util_test:verify_file_checksums_test2/0 (test/machi_admin_util_test.erl, line 48)
**exit:{not_dead_yet,<0.10485.0>}

Heh, I see fewer than 14 test failuers ... I see 6. Or 7, depending on the run. ``` *failed* in function machi_proxy_flu1_client_test:api_smoke_test/0 (test/machi_proxy_flu1_client_test.erl, line 72) **error:{badmatch,{error,partition}} machi_flu1_test: flu_smoke_test...*failed* in function machi_flu1_test:flu_smoke_test/0 (test/machi_flu1_test.erl, line 124) **error:{badmatch,{ok,{[{"prefix!^^00000000^e31aae63-9f81-46a0-9baa-60c66b72d7b0^0",0, <<"yo!">>, <<2,139,98,163,106,247,61,129,61,...>>}], []}}} machi_file_proxy_eqc:41: eqc_test_...*failed* in function eqc:quickcheck/1 (../src/eqc.erl, line 1270) in call from machi_file_proxy_eqc:'-eqc_test_/0-fun-1-'/1 (test/machi_file_proxy_eqc.erl, line 41) **exit:{badarg,[{eleveldb,async_get, [#Ref<0.0.0.40751>,<<>>, <<16,0,0,0,2,12,179,90,109,150,...>>, []], []}, {eleveldb,get,3,[{file,"src/eleveldb.erl"},{line,152}]}, {machi_csum_table,is_file_trimmed,2, [{file,"src/machi_csum_table.erl"},{line,158}]}, {machi_file_proxy,init,1, [{file,"src/machi_file_proxy.erl"},{line,224}]}, {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]} .*failed* in function eqc:quickcheck/1 (../src/eqc.erl, line 1270) in call from machi_ap_repair_eqc:'-prop_repair_test_/0-fun-1-'/3 (test/machi_ap_repair_eqc.erl, line 93) **exit:{timeout, {gen_server,call, [<0.10322.0>, {req, {append_chunk_extra, {1130,<<225,248,169,11,186,142,...>>}, [],0,<<"pre">>, <<2,47,38,147,...>>, 0}}, 2000]}} machi_ap_repair_eqc:101: prop_repair_par_test_...*failed* in function eqc:quickcheck/1 (../src/eqc.erl, line 1270) in call from machi_ap_repair_eqc:'-prop_repair_par_test_/0-fun-1-'/3 (test/machi_ap_repair_eqc.erl, line 103) **exit:{timeout, {gen_server,call, [<0.11008.0>, {req, {append_chunk_extra, {1115,<<168,193,7,104,172,160,...>>}, [],0,<<"pre">>, <<169,152,13,55,...>>, 0}}, 2000]}} [done in 18.022 s] module 'machi_admin_util_test' machi_admin_util_test: verify_file_checksums_test_...*failed* in function machi_util:wait_for_death/2 (src/machi_util.erl, line 357) in call from machi_test_util:start_flu_packages/1 (test/machi_test_util.erl, line 56) in call from machi_admin_util_test:verify_file_checksums_test2/0 (test/machi_admin_util_test.erl, line 48) **exit:{not_dead_yet,<0.10485.0>} ```

slfritchie commented

2016-03-29 09:29:22 +00:00

(Migrated from github.com)

This PR is stalled, alas, until staff is available to resume the unfinished work.

This pull request is broken due to missing fork information.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin unknown repository:unknown repository

git checkout unknown repository

Merge

Merge the changes and update on Forgejo.

git checkout master

git merge --no-ff unknown repository

git checkout master

git merge --ff-only unknown repository

git checkout unknown repository

git rebase master

git checkout master

git merge --no-ff unknown repository

git checkout master

git merge --squash unknown repository

git checkout master

git merge --ff-only unknown repository

git checkout master

git merge unknown repository

git push origin master

[WIP] Unify LevelDB usage to single instance #57

Checkout

Merge