[erlang-questions] fallocate(FALLOC_FL_PUNCH_HOLE) / lseek(SEEK_HOLE / SEEK_DATA) anytime soon ?

[Bernard Fouché]

Hi Richard,

thanks for your answer

Le 19/11/2012 22:04, Richard O'Keefe a écrit :
> On 20/11/2012, at 6:21 AM, Bernard Fouché wrote:
>
>> Hi List,
>>
>> Is there any chance that the API support these features in a reasonable time?
> Out of three operating systems immediately available to me,
> two (Solaris, Mac OS X) do not have fallocate, and one (three different
> versions of Linux that I am counting as one) does have fallocate() but
> insists that it has only an FALLOC_FL_KEEP_SIZE flag.
man 2 fallocate reports the same for me on Fedora 17 however 
FALLOC_FL_PUNCH_HOLE is listed in /usr/include/linux/falloc.h . I guess 
the man page is obsolete since kernel 2.6.38: 
http://man7.org/tlpi/api_changes/index.html#Linux-2.6.38 . It seems that 
Solaris has fcntl(F_FREESP) which provides the same feature. I can't 
tell about Mac OS X. The feature is available in more and more fs (zfs, 
xfs, etx4, btrfs and others).

With large files (2^64), in my use case (fixed size keys and fixed size 
values), I should be able to avoid indexing a table since I can lseek to 
(item_number * item_size) and free the corresponding disk block(s) when 
I don't need the data anymore, instead of managing a list of free blocks 
or instead of merging different files that handles deleted data by 
marking it with a tombstone value. Data will be fragmented but with a 
copy-on-write fs it is anyway and I guess it will be faster than to 
process an index or merging files. BTW some of the mentioned fs also 
have a defragment tool that can run on a live system.

> At its best, fallocate is Linux-specific, and posix_fallocate has
> no 'mode' argument, just fd, offset, and size.
> All of these operating systems support very large files, and
> also support files with holes where nothing was ever created in the
> first place.
>
> There is clearly room for Erlang access to operating-system-specific
> features, through operating-system-specific modules.
>
I'm still an Erlang novice and didn't look yet at the details of the 
interface with the underlying OS. I'll try to find how an IoDevice is 
matched to a fd...

   Bernard