File writing and odd behavior when unrelated IO activity occurs

[Rob Elsner]

All,

I'm recording a multicast stream using gen_udp in active mode (so my
process receives a mailbox message every time a packet arrives).  This
is for a 35 megabit/second stream.  I have a simple worker process
doing the recording:

loop(FileHandle, Socket, MulticastAddress, MulticastPort,
SourceAddress, FilePrefix) ->
	receive
		{udp, _, _, _, Bin} ->
			case file:write(FileHandle, Bin) of
				ok ->
					recorder:loop(FileHandle, Socket, MulticastAddress,
MulticastPort, SourceAddress, FilePrefix);
				{error, Reason} ->
					close_file(FileHandle), % close our current file
					close_socket(Socket, MulticastAddress, SourceAddress),
					log4erl:error("file:write returned error ~p", [Reason])
			end;
		{mark} ->
			NewName = get_file_name(MulticastAddress, MulticastPort,
SourceAddress, FilePrefix), % figure out the new file name based on
address and time
			log4erl:info("Recording mark ~p ~p ~p New name ~p",
[MulticastAddress, MulticastPort, SourceAddress, NewName]),
			{ok,NewHandle} = open_file(NewName), % open it
			close_file(FileHandle), % close our current file
			recorder:loop(NewHandle, Socket, MulticastAddress, MulticastPort,
SourceAddress, FilePrefix); % keep writing!
		{stop} ->
			close_file(FileHandle), % close our current file
			close_socket(Socket, MulticastAddress, SourceAddress)
		after 10000 ->
			log4erl:warn("Timed out ~p ~p ~p
~p~n",[MulticastAddress,MulticastPort,SourceAddress,FilePrefix]),
			close_file(FileHandle), % close our current file
			close_socket(Socket, MulticastAddress, SourceAddress)
	end.

These files get segmented at around 13GB for an hour long recording.
The issue I am seeing is that using the above process, if I delete a
file which has already been recorded, it causes the file:write process
to "hang" which causes my mailbox to fill up and consume all 8GB of
RAM on this box, before crashing the VM with an OOM error.

I've tried the file open options with raw set and not set, with
delayed_write.  Does anyone have any thoughts about why a file delete
operation on an unrelated file would cause file:write to block or take
an excessive amount of time? I've tried ionice'ing the erlang VM to
realtime highest priority, to no avail.

Thanks,
Rob Elsner