Update read_chunk() PB protocol to return trimmed chunks

kuenishi commented

2015-10-22 14:14:14 +00:00

(Migrated from github.com)

Add trimed array of chunk_pos at read_chunk message
trimmed are only filled only when needs_trimmed is true which is false by default
Pass options defined in PB protocol to the flu server
several minor changes in repair code (like change from spawn to spawn_link)

Will follow with actual trim implementation later.

- Add `trimed` array of chunk_pos at read_chunk message - `trimmed` are only filled only when `needs_trimmed` is true which is false by default - Pass options defined in PB protocol to the flu server - several minor changes in repair code (like change from `spawn` to `spawn_link`) Will follow with actual trim implementation later.

kuenishi commented

2015-10-23 07:16:31 +00:00

(Migrated from github.com)

Discussion result:

checksum must be recalculated after any trim
even if in read_chunk slicing case ( #27 )

So, next comes...

Basic actual trim & GC implementation
Checksum recalculation
(nice to have) TCP code separation from flu server

Discussion result: - checksum must be recalculated after any trim - even if in read_chunk slicing case ( #27 ) So, next comes... - Basic actual trim & GC implementation - Checksum recalculation - (nice to have) TCP code separation from flu server

kuenishi commented

2015-10-23 08:08:36 +00:00

(Migrated from github.com)

Currently the border trimming happens in CR client, because this makes repair life easier. This would be a point for discussion, maybe it'd better avoid repairing a whole GB chunk by reading a single byte.

slfritchie commented

2015-10-23 08:10:02 +00:00

(Migrated from github.com)

乙！+1, many thanks!

kuenishi commented

2015-10-23 08:11:54 +00:00

(Migrated from github.com)

@slfritchie Thank you for review, btw what do you think of slicing the chunk returned by flu to fit client-requested size at CR client?

slfritchie commented

2015-10-23 09:26:02 +00:00

(Migrated from github.com)

If the original chunk was 1MB and the client asked for 1KB in the middle of the original chunk, why should the server send 1023 extra KB? (The "reductio ad absurdum" https://en.wikipedia.org/wiki/Reductio_ad_absurdum argument suggests that for any read_chunk request, the server should send the whole file ... send all files ... send the entire Internet! ^_^)

I like the idea of making the clients to as much "extra work" as is reasonable ... but I feel that sending the original chunk (send the entire Internet) isn't a good use of network bandwidth: the server memory & CPU is cheaper than network b/w?

If the original chunk was 1MB and the client asked for 1KB in the middle of the original chunk, why should the server send 1023 extra KB? (The "reductio ad absurdum" https://en.wikipedia.org/wiki/Reductio_ad_absurdum argument suggests that for any read_chunk request, the server should send the whole file ... send all files ... send the entire Internet! `^_^`) I like the idea of making the clients to as much "extra work" as is reasonable ... but I feel that sending the original chunk (send the entire Internet) isn't a good use of network bandwidth: the server memory & CPU is cheaper than network b/w?

kuenishi commented

2015-10-23 09:51:14 +00:00

(Migrated from github.com)

@slfritchie That makes sense. I pushed the fix at #27.

Update read_chunk() PB protocol to return trimmed chunks #25