[erlang-questions] Reading a file before it has been completely written

[Daniel Luna]

(Sorry, Richard for getting this multiple times, forgot to include the list)

Just as an added caveat to what Richard mentions: you probably want to
make sure that the temp file is created on the same file system as
your test directory.  A rename between file systems is really just a
copy and delete, with the exact same problems you had previously
(depending on OS I guess), while a rename within a file system is a
link/unlink with the contents being untouched.

/Daniel

On 7 March 2012 12:54, Richard Carlsson <carlsson.richard@REDACTED> wrote:
> On 03/07/2012 06:25 PM, David Mercer wrote:
>>
>> While this isn’t an Erlang-specific question, the problem arises from my
>> using Richard Carlsson’s /file_monitor/
>>
>> (https://github.com/richcarl/eunit/blob/master/src/file_monitor.erl),
>> which sends messages when a file or directory is changed. I have found
>> that it is not unusual to get a message about a new file before the file
>> has been completely written.
>>
>> I had thought that by doing a file:open(Filepath, [read]) and making
>> sure I got back {ok, _} rather than {error, eacces} I could avoid those
>> cases, but that approach has failed for me: this morning, I got back
>> {ok, _}, but the file was not completely written yet.
>>
>> Another approach I tried was to attempt to obtain an exclusive lock (I
>> think it was file:open(Filepath, [read, exclusive])), but in my testing
>> I came across the bizarre scenario where I would copy a file into the
>> monitored directory, the file_monitor would send the message, but the
>> Erlang process that does the file-open didn’t see it, so created the
>> file (the documentation says it creates the file if it does not exist),
>> and then I got a message in my window where I was copying that the file
>> already exists, do I want to overwrite it.
>>
>> Another approach I tried was renaming the file to itself. All my tests
>> indicated that that approach would work, but all my tests also indicated
>> that just doing the file:open(Filepath, [read]) would work, too, so I
>> chose it, as it seemed cleaner. I could revert to the rename approach,
>> but I’m not even sure now that that will work.
>>
>> I imagine others among us have encountered this issue, and rather than
>> reinvent the wheel, what is the favored approach to handling this issue?
>>
>> Cheers,
>>
>> David Mercer
>
>
> Hey, a user! I haven't had any reports about this module before (and the
> fact that it's still in my development branch of eunit is more of a
> historical accident; it's not shipped with OTP). I don't know of any real
> issues with it though.
>
> In this case, I think the problem is just the underlying file system
> semantics. I presume it's Linux, and in Unix:y file systems a file can be
> seen to exist and can be opened for reading as soon as it has been created.
> Trying to fiddle with exclusive locks is probably always going to have
> corner cases. The only techniques you can trust to practically always work
> and be portable across file systems are directory creation and file
> renaming. So what Tony suggested is likely to be the best solution: create
> the file under another name or in a separate directory, and when it's
> completely written, rename it.
>
>   /Richard
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions