Commit graph

509 commits

Author SHA1 Message Date
Shunichi Shinohara
eef00e4f8f Add TODO comment for possible race condition 2015-12-10 15:58:17 +09:00
Shunichi Shinohara
2e2d282afc Use outside of ephemeral port range to listen on
When there is TCP_WAIT connection whose local part has
port to be listened, listen (bind) will fail by eaddrinuse
_on Linux_ (won't on Mac OS X).
This commit also adds some logs and pattern matches.

Reference
- Ephemeral port - Wikipedia, the free encyclopedia
  https://en.wikipedia.org/wiki/Ephemeral_port

  "Many Linux kernels use the port range 32768 to 61000.[note 2]
  FreeBSD has used the IANA port range since release 4.6. Previous
  versions, including the Berkeley Software Distribution (BSD), use
  ports 1024 to 5000 as ephemeral ports.[2]"

- Demostration of collision between already-closed ephemeral port
  and listen port on Linux (Mac OS X allows)
  https://gist.github.com/shino/36ae1e01608366d52236
2015-12-09 18:04:50 +09:00
Shunichi Shinohara
ade4430d30 More cleaner clean up 2015-12-09 10:29:45 +09:00
Shunichi Shinohara
aa0a0413d1 Cosmetics of comments, specs, whitespaces and unit tests refactoring 2015-12-09 09:58:34 +09:00
Shunichi Shinohara
14765a7279 Change ranch callback module name 2015-12-09 09:58:34 +09:00
Shunichi Shinohara
a1f5a6ce62 Fix unit test cases around flu1 startup 2015-12-09 09:58:34 +09:00
Shunichi Shinohara
7614910f36 Initialize FLU package with ranch listener 2015-12-09 09:58:33 +09:00
Shunichi Shinohara
9579b1b8b2 Introduce ranch and add transport callback 2015-12-09 09:58:33 +09:00
Shunichi Shinohara
83b4466a85 Merge pull request #50 from basho/slf/doc-name-game2
Review & merge slf/doc-name-game2
2015-12-09 09:45:11 +09:00
UENISHI Kota
293eb4810f Fix dialyzer error 2015-12-07 14:49:34 +09:00
UENISHI Kota
89e80a8862 Fix GC not running 2015-12-07 12:07:46 +09:00
UENISHI Kota
07c2b97918 Change checksum_list API to return a t2b list 2015-12-07 10:55:45 +09:00
UENISHI Kota
befa776685 Fix several new bugs 2015-12-04 17:38:36 +09:00
UENISHI Kota
8528567954 Add eleveldb with sext to use it as metadata storage
First step is to use as checksum table. It will also used for file
names store and *ALL* other persistent metadata than files.
2015-12-04 16:38:57 +09:00
Scott Lystig Fritchie
35c48300a5 Fix Dialyzer complaints, derp! 2015-12-04 15:21:44 +09:00
Scott Lystig Fritchie
e9b1134cd9 Merge pull request #48 from basho/mra/merkle-cleanup
Add merkle library
2015-12-02 16:25:50 +09:00
Scott Lystig Fritchie
d44e9dd542 Fix plumbing for find_file() 2015-12-02 15:54:34 +09:00
Scott Lystig Fritchie
916ac754d7 WIP: still broken, almost passes suites=machi_cr_client_test tests=smoke_test_ 2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
5477f3b6f8 WIP broken 2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
c3002d9852 Push TODO closer to actual TODO work site, standardize var spelling 2015-12-02 15:35:39 +09:00
Scott Lystig Fritchie
10a27ce7dd All eunit tests now passing again 2015-12-02 15:35:38 +09:00
Scott Lystig Fritchie
014ba89e3a Add client-side plumbing for high proto append chunk CoC 2015-12-02 15:35:38 +09:00
Mark Allen
a5a0369905 Add basic unit test 2015-11-20 21:22:41 -06:00
Mark Allen
4ce7a87d56 Remove merklet 2015-11-20 16:29:17 -06:00
UENISHI Kota
6786820401 Merge pull request #35 from basho/ku/making-file-proxy-spec
Add eqc trim tests to machi_file_proxy
2015-11-05 16:27:48 +09:00
Shunichi Shinohara
1b0711f151 Stop flu1 client under CR client 2015-11-05 11:46:22 +09:00
UENISHI Kota
d0e6417f5d Remove unnecessary output 2015-11-04 16:15:33 +09:00
UENISHI Kota
3f6f9e2c6b Address one offset+length issue 2015-11-04 16:08:43 +09:00
UENISHI Kota
c1e5426034 Address PR comments 2015-11-04 16:08:09 +09:00
UENISHI Kota
3b087c0388 Add eqc trim tests to machi_file_proxy
* Add description on high client APIs
* Add notes to rethink high client specification
2015-11-04 16:02:29 +09:00
Shunichi Shinohara
059f591d3f Exclude FLUs which can not send to cr client as well as receive from
in terms of partition simulator
2015-11-02 17:57:16 +09:00
Mark Allen
3c5a9e6f53 Torture tests for merkle tree
1,000,000 entries - timings and size
2015-11-02 00:12:58 -06:00
Mark Allen
72a4fab49d Add a naive diff function 2015-10-29 22:18:20 -05:00
Shunichi Shinohara
6fa2de28cd Add self pid to debug print 2015-10-30 09:44:54 +09:00
Shunichi Shinohara
93b168415d Change log level to debug for multiple files with certain seq num 2015-10-30 09:40:07 +09:00
Shunichi Shinohara
bf5768eb47 Make CR client partition-simulator-aware 2015-10-30 09:39:21 +09:00
Shunichi Shinohara
447c8c8d48 Add 2-tuple timeout setting to CR client 2015-10-30 09:39:21 +09:00
Scott Lystig Fritchie
611f33e81b Change trigger_gc default -> 0 2015-10-29 15:05:15 +09:00
UENISHI Kota
028135d927 Update some comments for concise and sound description 2015-10-29 12:07:34 +09:00
Mark Allen
7086899941 Reorg merkle tree code into a library
Was a service previously. Now contains both merklet
and the naive implementations.  Put construction
timing stuff into the test.

Tests are not truly meaningful yet.
2015-10-28 16:59:49 -05:00
UENISHI Kota
170b3cd797 Dialyzer fix 2015-10-28 12:48:50 +09:00
UENISHI Kota
f7358424e4 Trim command and GC prototype implementation
* maybe_gc/2 is triggered at machi_file_proxy, when chunk is deleted
  and the file is larger than `max_file_size`
* A file is deleted if all chunks except 1024 bytes header are trimmed
* If a file is going to be deleted, file_proxy notifies metadata_mgr
  to remember the filename persistently, whose filename is
  `known_files_<FluName>`
* Such trimmed filenames are stored in a machi_plist file per flu
* machi_file_proxy could not be started if the filename is in the
  manager's list. Consequently, any write, read and trim operations
  cannot happen against deleted file.
* After the file was trimmed, any read request to the file returns
  `{error, trimmed}`
* Disclaimer: no tests written yet and machi_plist does not support
  any recovery from partial writes.
* Add some thoughts as comments for repairing trims.

* State diagram of every byte is as follows:

```
state\action| write/append   | read_chunk       | trim_chunk
------------+----------------+------------------+---------------
 unwritten  |  -> written    | fail (+repair)   | -> trimmed
 written    | noop or repair | return content   | -> trimmed
 trimmed    |  fail          | fail             | noop
```
2015-10-28 12:34:03 +09:00
Mark Allen
5e571f6009 Switch to merklet
Still a WIP
2015-10-27 16:33:18 -05:00
Mark Allen
b710517c64 Fixes after testing 2015-10-27 11:57:38 -05:00
Mark Allen
48fabdcd5f WIP
Almost certainly broken
2015-10-27 11:57:38 -05:00
Scott Lystig Fritchie
61f02dfc9f Merge pull request #30 from basho/ss-fix-opt-timeout-arg-mismatch
Fix missing "options" arg for a few calls
2015-10-27 14:36:52 +09:00
Scott Lystig Fritchie
5ec35773e3 Merge pull request #28 from basho/ku/csum-table-anybytes
Change machi_csum_table to support arbitrary bytes writes and trims
2015-10-27 14:21:09 +09:00
Scott Lystig Fritchie
d2b1c7512a Merge branch 'ku/config-system' into tmp 2015-10-27 14:14:40 +09:00
Scott Lystig Fritchie
b500d5f449 config hell 2015-10-27 14:07:45 +09:00
Scott Lystig Fritchie
dd17b1de0a Remove compiler warnings 2015-10-27 14:07:07 +09:00