Remember eofp of all files #47

New issue

Open

opened 2015-11-20 08:06:18 +00:00 by kuenishi · 2 comments

kuenishi commented

2015-11-20 08:06:18 +00:00

(Migrated from github.com)

Below are quote from Scott's mail:

Background:

In the append operation, the client can request an allocation of extra space, e.g. the client wants to write 500MB eventually but only has 64KB now.  The client can do something like:

1. {ok, File, Offset} = append(Prefix, Chunk64K, Extra=(500MB-64KB))
2. read next 64K chunk
3. write(File, Offset+64K, Chunk64K_2)
4. goto 2 and continue advancing the offset until all 500MB are written

Clarification:

The FLU's sequencer must not forget any extra space allocations requested by a client.  The client can issue those write() ops much later, e.g. a month later.

Thoughts?

During #45 work it was revealed that eofp in file proxy will be forgotten after reboot of file proxy or whole flu server process, which should be remembered. This is problematic especially when `chunk_extra` was given by client and reserved for later, but client is so slow that file proxy process "sleeps" by stopping. My idea is to store them in eleveldb, which makes eleveldb introduction priority higher than before. Below are quote from Scott's mail: ``` Background: In the append operation, the client can request an allocation of extra space, e.g. the client wants to write 500MB eventually but only has 64KB now. The client can do something like: 1. {ok, File, Offset} = append(Prefix, Chunk64K, Extra=(500MB-64KB)) 2. read next 64K chunk 3. write(File, Offset+64K, Chunk64K_2) 4. goto 2 and continue advancing the offset until all 500MB are written Clarification: The FLU's sequencer must not forget any extra space allocations requested by a client. The client can issue those write() ops much later, e.g. a month later. ``` Thoughts?

slfritchie commented

2015-11-20 08:10:06 +00:00

(Migrated from github.com)

Agreed. We have more persistence options than eleveldb, but eleveldb is the most convenient.

jadeallenx commented

2015-11-20 16:30:48 +00:00

(Migrated from github.com)

Good clarification. I didn't understand that requirement from our previous discussions. Yeah sequencer and filename state are increasingly important to proper system operation.