[erlang-questions] Reading a file before it has been completely written

[Richard Carlsson]

On 03/07/2012 06:25 PM, David Mercer wrote:
> While this isn’t an Erlang-specific question, the problem arises from my
> using Richard Carlsson’s /file_monitor/
> (https://github.com/richcarl/eunit/blob/master/src/file_monitor.erl),
> which sends messages when a file or directory is changed. I have found
> that it is not unusual to get a message about a new file before the file
> has been completely written.
>
> I had thought that by doing a file:open(Filepath, [read]) and making
> sure I got back {ok, _} rather than {error, eacces} I could avoid those
> cases, but that approach has failed for me: this morning, I got back
> {ok, _}, but the file was not completely written yet.
>
> Another approach I tried was to attempt to obtain an exclusive lock (I
> think it was file:open(Filepath, [read, exclusive])), but in my testing
> I came across the bizarre scenario where I would copy a file into the
> monitored directory, the file_monitor would send the message, but the
> Erlang process that does the file-open didn’t see it, so created the
> file (the documentation says it creates the file if it does not exist),
> and then I got a message in my window where I was copying that the file
> already exists, do I want to overwrite it.
>
> Another approach I tried was renaming the file to itself. All my tests
> indicated that that approach would work, but all my tests also indicated
> that just doing the file:open(Filepath, [read]) would work, too, so I
> chose it, as it seemed cleaner. I could revert to the rename approach,
> but I’m not even sure now that that will work.
>
> I imagine others among us have encountered this issue, and rather than
> reinvent the wheel, what is the favored approach to handling this issue?
>
> Cheers,
>
> David Mercer

Hey, a user! I haven't had any reports about this module before (and the 
fact that it's still in my development branch of eunit is more of a 
historical accident; it's not shipped with OTP). I don't know of any 
real issues with it though.

In this case, I think the problem is just the underlying file system 
semantics. I presume it's Linux, and in Unix:y file systems a file can 
be seen to exist and can be opened for reading as soon as it has been 
created. Trying to fiddle with exclusive locks is probably always going 
to have corner cases. The only techniques you can trust to practically 
always work and be portable across file systems are directory creation 
and file renaming. So what Tony suggested is likely to be the best 
solution: create the file under another name or in a separate directory, 
and when it's completely written, rename it.

    /Richard