Lost data when sending to a linked in driver

[Shawn Pearce]

I'm using R8B-1 and a custom linked in driver that I'm currently developing.
The driver uses the outputv hook to receive binary data sent by the Erlang
node.  When I'm sending data to the driver, I use erlang:port_call to set
the driver into some state, at which point it receives data through the
outputv hook (using Port ! {self(), {command, SomeList}}).

SomeList is a proper Erlang list of 32 binaries, each 1.5 MB in size.

Erts decides to send 1 element in the ErlIOVec which is passed to my
driver's outputv hook.  This element is the customary 0th "empty"
element for the driver to hook in any header data prior to transmission.
None of the binaries I sent to the driver from Erlang were presented to
the driver.

However, if I send a list of only 15 binaries, each 1.5 MB in size
they all appear in the ErlIOVec.  The driver gets 16 items in the
ErlIOVec, with the 15 binaries supplied starting at index 1, as
they should.

Unfortunately, there seems to be no way for Erlang or the driver to
know that Erts dropped the list of binaries on the floor when it
exceeds 15 elements.  If these binaries were smaller, I'd expect
that Erts would compact them into a single binary, but they aren't
(and I can't shrink them, its my data's natural chunk size).
I also don't really want Erts copying 32 MB of data into a compacted
binary here.  Zero-copy needs to be the name of the game. :-)

So I've come up with this twisted chunk of code for the port driver
to send to my linked in driver:

	% Taken from erts/emulator/beam/io.c: SMALL_WRITE_VEC = 16
	% This is how many binaries we can send down.  io.c adds one binary on top
	% of this in case the driver wants to use it to define a packet header.
	% This header slot is unused, giving us 15 buffers.
-define(MAX_IOVEC, 15).

post_frames({Port}, FrameList) ->
    post_frames(Port, FrameList, [], 0).
	      
post_frames(_, [], [], 0) ->
	ok;
post_frames(Port, [], Bundle, Cnt) ->
	Port ! {self(), {command, lists:reverse(Bundle)}},
	receive
		{Port, frames_queued, Count} ->
			ok
	end;
post_frames(Port, FrameList, Bundle, Cnt) when Cnt == ?MAX_IOVEC ->
	Port ! {self(), {command, lists:reverse(Bundle)}},
	receive
		{Port, frames_queued, Count} ->
			post_frames(Port, FrameList, [], 0)
	end;
post_frames(Port, [Frame | List], Bundle, Cnt) ->
	post_frames(Port, List, [Frame | Bundle], Cnt + 1).


This code however, screams out to me as having a few issues.  The first
is that MAX_IOVEC is set deep down inside of the emulator.  There is
no way to get this value from Erlang at runtime, or from the driver
at compile time or runtime.  If SMALL_WRITE_VEC ever changed in the
eumulator from 16 to another value, this code could break.

The second issue is that when creating the list of frames which can
be sent to the driver, I need to compose that list, then reverse it.
Would it be better if I wrote the post_frames/4 function more like:

post_frames(Port, [F1,F2,F3,F4,F5,F6,F7,F8 | T]) ->
	Port ! {self(), {command, [F1,F2,F3,F4,F5,F6,F7,F8]}},
	receive
		{Port, frames_queued, Count} ->
			post_frames(Port, T)
	end;
post_frames(Port, [F1,F2,F3,F4 | T]) ->
	Port ! {self(), {command, [F1,F2,F3,F4]}},
	receive
		{Port, frames_queued, Count} ->
			post_frames(Port, T)
	end;
post_frames(Port, [F1,F2 | T]) ->
	Port ! {self(), {command, [F1,F2]}},
	receive
		{Port, frames_queued, Count} ->
			post_frames(Port, T)
	end;
post_frames(Port, [F1 | T]) ->
	Port ! {self(), {command, [F1,F2]}},
	receive
		{Port, frames_queued, Count} ->
			post_frames(Port, T)
	end;
post_frames(Port, []) ->
	ok.

as this version now does not require creating two lists, and reversing one of
them?  Unfortunately, it still suffers from the problem that if the value of
SMALL_WRITE_VEC ever changed from 16 to smaller than 8 I could run the risk
of losing all 8 binaries when they are matched in the first clause.

I would really like to be able to send a group of binaries to the driver
at once, as the driver is using locking a pthread mutexes directly to
interface to some "legacy" pthread code, delivering the group of
binaries to that "legacy" pthread (after incremeting ErlDrvBinary.refc.

The cost of sending one binary is about the same as sending 8 or 12 binaries
at a time, due to the cost of locking and unlocking a pthread mutex, so I'd
like to 'batch' them as much as possible, especially since the port driver
will be receiving batches of binaries from the rest of the Erlang system.

At any rate, no matter how I implement my code, it looks like there might
be a bug in io.c when the number of binaries exceeds 15 and their size is
too large to compact them.

--
Shawn.

Why do I like Erlang?  Because ``it'll drop your data for you, without
asking you.''  (See above email.  :-)

Why do I like Perl?  Because ``in accordance with Unix tradition Perl
gives you enough rope to hang yourself with.''

Why do I dislike Java? Because ``the class ROPE that should contain the
method HANG to do the hanging doesn't exist because there is too much
'security' built into the base language.''