Trim command and GC prototype implementation

kuenishi commented

2015-10-28 03:35:28 +00:00

(Migrated from github.com)

maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
and the file is larger than max_file_size
A file is deleted if all chunks except 1024 bytes header are trimmed
If a file is going to be deleted, file_proxy notifies metadata_mgr
to remember the filename persistently, whose filename is
known_files_<FluName>
Such trimmed filenames are stored in a machi_plist file per flu
machi_file_proxy could not be started if the filename is in the
manager's list. Consequently, any write, read and trim operations
cannot happen against deleted file.
After the file was trimmed, any read request to the file returns
{error, trimmed}
Disclaimer: no tests written yet and machi_plist does not support
any recovery from partial writes.
Add some thoughts as comments for repairing trims.
State diagram of every byte is as follows:

state\action| write/append   | read_chunk       | trim_chunk
------------+----------------+------------------+---------------
 unwritten  |  -> written    | fail (+repair)   | -> trimmed
 written    | noop or repair | return content   | -> trimmed
 trimmed    |  fail          | fail             | noop

This pull request is supposed to cover all thoughts or fix to the issues mentioned in #14.

- maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted and the file is larger than `max_file_size` - A file is deleted if all chunks except 1024 bytes header are trimmed - If a file is going to be deleted, file_proxy notifies metadata_mgr to remember the filename persistently, whose filename is `known_files_<FluName>` - Such trimmed filenames are stored in a machi_plist file per flu - machi_file_proxy could not be started if the filename is in the manager's list. Consequently, any write, read and trim operations cannot happen against deleted file. - After the file was trimmed, any read request to the file returns `{error, trimmed}` - Disclaimer: no tests written yet and machi_plist does not support any recovery from partial writes. - Add some thoughts as comments for repairing trims. - State diagram of every byte is as follows: ``` state\action| write/append | read_chunk | trim_chunk ------------+----------------+------------------+--------------- unwritten | -> written | fail (+repair) | -> trimmed written | noop or repair | return content | -> trimmed trimmed | fail | fail | noop ``` This pull request is supposed to cover all thoughts or fix to the issues mentioned in #14.

kuenishi commented

2015-10-28 03:39:18 +00:00

(Migrated from github.com)

I should revisit race condition matrix of a byte including repair requests with more solid tests, which is a huge todo.

kuenishi commented

2015-10-29 02:32:31 +00:00

(Migrated from github.com)

So I added TRIMMED flag to PB protocol so that read_chunk can return {error, trimmed} which means all chunks of the file was trimmed. The background here are some combination of my thoughts:

Maybe, returning {ok, {[], [{0, infinity}]}} is the very same response but an atom infinity is a more wary value, which is often rather complicated to describe in other languages, like PB IDL. Just defining integer is much simpler.
In current API semantics where we already exposed the notion of files, applications that use Machi must remember all file names assigned by Machi, to read chunks. If Machi tells them that whole file was deleted, they'll write a code like case read_chunk(...) of {error, trimmed} -> { forget about that file }; {ok, {Chunks, _}} -> { do what I wanted } . This would be easier than to find out infinity from the response.
That also matches internal behaviour, where a special repair different from repairing chunks but share trimmed state of that file among flu servers, might be needed.

So I added `TRIMMED` flag to PB protocol so that `read_chunk` can return `{error, trimmed}` which means all chunks of the file was trimmed. The background here are some combination of my thoughts: 1. Maybe, returning `{ok, {[], [{0, infinity}]}}` is the very same response but an atom `infinity` is a more wary value, which is often rather complicated to describe in other languages, like PB IDL. Just defining integer is much simpler. 2. In current API semantics where we already exposed the notion of files, applications that use Machi must remember all file names assigned by Machi, to read chunks. If Machi tells them that whole file was deleted, they'll write a code like `case read_chunk(...) of {error, trimmed} -> { forget about that file }; {ok, {Chunks, _}} -> { do what I wanted }` . This would be easier than to find out `infinity` from the response. 3. That also matches internal behaviour, where a special repair different from repairing chunks but share trimmed state of that file among flu servers, might be needed.

slfritchie commented

2015-10-29 05:59:12 +00:00

(Migrated from github.com)

Looks like we have some disagreement about the API & trimmed cases. We don't have a customer yet, so ... let's punt until we are closer to a real use case.

slfritchie commented

2015-10-29 06:07:08 +00:00

(Migrated from github.com)

Thanks! +1

kuenishi commented

2015-10-29 06:18:55 +00:00

(Migrated from github.com)

Next step:

Improve file_proxy EQC tests to cover cross-border trims and reads
Add EQC tests to csum_table
Persistent storage of metadatas (how about SQLite for example)
Background process (repairor) design
Reparing trimmed files when it received {error, trimmed}
Improve partition tests to include trim command
Write initial draft API documentation to machi_pb_high_client.erl.

Next step: - Improve file_proxy EQC tests to cover cross-border trims and reads - Add EQC tests to csum_table - Persistent storage of metadatas (how about SQLite for example) - Background process (repairor) design - Reparing trimmed files when it received `{error, trimmed}` - Improve partition tests to include `trim` command - Write initial draft API documentation to `machi_pb_high_client.erl`.

Trim command and GC prototype implementation #32