Commit graph

19 commits

Author SHA1 Message Date
Scott Lystig Fritchie
c49ccafdc6 Merge slf/flu-config-rcd-style 2015-12-18 15:41:02 +09:00
Scott Lystig Fritchie
0fc7bc74b7 EDoc fixes 2015-12-08 22:05:11 +09:00
UENISHI Kota
293eb4810f Fix dialyzer error 2015-12-07 14:49:34 +09:00
UENISHI Kota
89e80a8862 Fix GC not running 2015-12-07 12:07:46 +09:00
UENISHI Kota
07c2b97918 Change checksum_list API to return a t2b list 2015-12-07 10:55:45 +09:00
UENISHI Kota
befa776685 Fix several new bugs 2015-12-04 17:38:36 +09:00
UENISHI Kota
8528567954 Add eleveldb with sext to use it as metadata storage
First step is to use as checksum table. It will also used for file
names store and *ALL* other persistent metadata than files.
2015-12-04 16:38:57 +09:00
UENISHI Kota
d0e6417f5d Remove unnecessary output 2015-11-04 16:15:33 +09:00
UENISHI Kota
170b3cd797 Dialyzer fix 2015-10-28 12:48:50 +09:00
UENISHI Kota
f7358424e4 Trim command and GC prototype implementation
* maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
  and the file is larger than `max_file_size`
* A file is deleted if all chunks except 1024 bytes header are trimmed
* If a file is going to be deleted, file_proxy notifies metadata_mgr
  to remember the filename persistently, whose filename is
  `known_files_<FluName>`
* Such trimmed filenames are stored in a machi_plist file per flu
* machi_file_proxy could not be started if the filename is in the
  manager's list. Consequently, any write, read and trim operations
  cannot happen against deleted file.
* After the file was trimmed, any read request to the file returns
  `{error, trimmed}`
* Disclaimer: no tests written yet and machi_plist does not support
  any recovery from partial writes.
* Add some thoughts as comments for repairing trims.

* State diagram of every byte is as follows:

```
state\action| write/append   | read_chunk       | trim_chunk
------------+----------------+------------------+---------------
 unwritten  |  -> written    | fail (+repair)   | -> trimmed
 written    | noop or repair | return content   | -> trimmed
 trimmed    |  fail          | fail             | noop
```
2015-10-28 12:34:03 +09:00
UENISHI Kota
7377624579 Dialyzer cleanup 2015-10-27 13:44:12 +09:00
UENISHI Kota
6e3347f727 Support log replay 2015-10-27 13:43:45 +09:00
UENISHI Kota
8a61055f55 Support arbitrary bytes write by using find_(left|right)neighbor/2 2015-10-27 13:43:45 +09:00
Scott Lystig Fritchie
b3b24b1178 Type corrections & remove dead code 2015-10-21 18:37:31 +09:00
Scott Lystig Fritchie
028ddc79ff Data type cleanups, other 2015-10-21 18:37:30 +09:00
Scott Lystig Fritchie
595f9a463e Unexported funcs 2015-10-21 18:37:30 +09:00
UENISHI Kota
ebb9bc3f5a Allow reading multiple chunks at once
* When repairing multiple chunks at once and any of its repair
  failed, the whole read request and repair work will fail
* Rename read_repair3 and read_repair4 to do_repair_chunks and
  do_repair chunk in machi_file_proxy
* This pull request changes return semantics of read_chunk(), that
  returns any chunk included in requested range
* First and last chunk may be cut to fit the requested range
* In machi_file_proxy, unwritten_bytes are removed and replaced by
  machi_csum_table
2015-10-20 17:59:09 +09:00
UENISHI Kota
6f790527f5 Follow with missing tests and related fix 2015-10-16 10:10:05 +09:00
UENISHI Kota
e45469b5ce Move checksum file related code to machi_csum_table 2015-10-15 11:28:40 +09:00