Possible disk space issues #49

New issue

Open

opened 2015-11-24 08:52:18 +00:00 by kuenishi · 0 comments

kuenishi commented

2015-11-24 08:52:18 +00:00

(Migrated from github.com)

Problems:

Clients die and lose assigned file names and chunks at append and they never are trimmed
Clients die and unwritten space assigned via chunk_extra via append never be written
Say, 20GB have 19.8 GB chunk trimmed and 0.2 GB written - consumes 20GB for 0.2GB data until it is trimmed

Possible resolutions:

Application-side garbage collection of listing all files that belongs to a prefix+namespace?
Same above?
Compaction or any other way to make that file as sparse file? fallocate(2) in Linux has FALLOC_FL_PUNCH_HOLE option so that we can kill trimmed space. Wow. I found an answer right after I wrote a question.

Problem 1, 2 are different from 3 as with FALLOC_FL_PUNCH_HOLE space are no more issue but overhead of keeping files in that way and never being collected, like inodes and so on.

Shortcoming of resolution 3 is that makes Machi Linux-specific. A simple googling made me feel that for Solaris and FreeBSD there might be an API for punching hole in file.

Problems: 1. Clients die and lose assigned file names and chunks at `append` and they never are trimmed 2. Clients die and unwritten space assigned via `chunk_extra` via `append` never be written 3. Say, 20GB have 19.8 GB chunk trimmed and 0.2 GB written - consumes 20GB for 0.2GB data until it is trimmed Possible resolutions: 1. Application-side garbage collection of listing all files that belongs to a prefix+namespace? 2. Same above? 3. Compaction or any other way to make that file as sparse file? [fallocate(2) in Linux](http://man7.org/linux/man-pages/man2/fallocate.2.html) has `FALLOC_FL_PUNCH_HOLE` option so that we can kill trimmed space. Wow. I found an answer right after I wrote a question. Problem 1, 2 are different from 3 as with `FALLOC_FL_PUNCH_HOLE` space are no more issue but overhead of keeping files in that way and never being collected, like inodes and so on. Shortcoming of resolution 3 is that makes Machi Linux-specific. A simple googling made me feel that for Solaris and FreeBSD there might be an API for punching hole in file.