Lost data when sending to a linked in driver

[Raimo Niskanen]

Sounds like a bug to me, I will look into it.

As a comfort, i guess that SMALL_WRITE_VEC will probably never become
less than 16. It is now set to match the least allowed value according
to Posix.1g of IOV_MAX in sys/uio.h.

/ Raimo Niskanen, Erlang/OTP, Ericsson AB



Shawn Pearce wrote:
> 
> I'm using R8B-1 and a custom linked in driver that I'm currently developing.
> The driver uses the outputv hook to receive binary data sent by the Erlang
> node.  When I'm sending data to the driver, I use erlang:port_call to set
> the driver into some state, at which point it receives data through the
> outputv hook (using Port ! {self(), {command, SomeList}}).
> 
> SomeList is a proper Erlang list of 32 binaries, each 1.5 MB in size.
> 
> Erts decides to send 1 element in the ErlIOVec which is passed to my
> driver's outputv hook.  This element is the customary 0th "empty"
> element for the driver to hook in any header data prior to transmission.
> None of the binaries I sent to the driver from Erlang were presented to
> the driver.
> 
> However, if I send a list of only 15 binaries, each 1.5 MB in size
> they all appear in the ErlIOVec.  The driver gets 16 items in the
> ErlIOVec, with the 15 binaries supplied starting at index 1, as
> they should.
> 
> Unfortunately, there seems to be no way for Erlang or the driver to
> know that Erts dropped the list of binaries on the floor when it
> exceeds 15 elements.  If these binaries were smaller, I'd expect
> that Erts would compact them into a single binary, but they aren't
> (and I can't shrink them, its my data's natural chunk size).
> I also don't really want Erts copying 32 MB of data into a compacted
> binary here.  Zero-copy needs to be the name of the game. :-)
> 
> So I've come up with this twisted chunk of code for the port driver
> to send to my linked in driver:
> 
>         % Taken from erts/emulator/beam/io.c: SMALL_WRITE_VEC = 16
>         % This is how many binaries we can send down.  io.c adds one binary on top
>         % of this in case the driver wants to use it to define a packet header.
>         % This header slot is unused, giving us 15 buffers.
> -define(MAX_IOVEC, 15).
> 
> post_frames({Port}, FrameList) ->
>     post_frames(Port, FrameList, [], 0).
> 
> post_frames(_, [], [], 0) ->
>         ok;
> post_frames(Port, [], Bundle, Cnt) ->
>         Port ! {self(), {command, lists:reverse(Bundle)}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         ok
>         end;
> post_frames(Port, FrameList, Bundle, Cnt) when Cnt == ?MAX_IOVEC ->
>         Port ! {self(), {command, lists:reverse(Bundle)}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         post_frames(Port, FrameList, [], 0)
>         end;
> post_frames(Port, [Frame | List], Bundle, Cnt) ->
>         post_frames(Port, List, [Frame | Bundle], Cnt + 1).
> 
> This code however, screams out to me as having a few issues.  The first
> is that MAX_IOVEC is set deep down inside of the emulator.  There is
> no way to get this value from Erlang at runtime, or from the driver
> at compile time or runtime.  If SMALL_WRITE_VEC ever changed in the
> eumulator from 16 to another value, this code could break.
> 
> The second issue is that when creating the list of frames which can
> be sent to the driver, I need to compose that list, then reverse it.
> Would it be better if I wrote the post_frames/4 function more like:
> 
> post_frames(Port, [F1,F2,F3,F4,F5,F6,F7,F8 | T]) ->
>         Port ! {self(), {command, [F1,F2,F3,F4,F5,F6,F7,F8]}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         post_frames(Port, T)
>         end;
> post_frames(Port, [F1,F2,F3,F4 | T]) ->
>         Port ! {self(), {command, [F1,F2,F3,F4]}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         post_frames(Port, T)
>         end;
> post_frames(Port, [F1,F2 | T]) ->
>         Port ! {self(), {command, [F1,F2]}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         post_frames(Port, T)
>         end;
> post_frames(Port, [F1 | T]) ->
>         Port ! {self(), {command, [F1,F2]}},
>         receive
>                 {Port, frames_queued, Count} ->
>                         post_frames(Port, T)
>         end;
> post_frames(Port, []) ->
>         ok.
> 
> as this version now does not require creating two lists, and reversing one of
> them?  Unfortunately, it still suffers from the problem that if the value of
> SMALL_WRITE_VEC ever changed from 16 to smaller than 8 I could run the risk
> of losing all 8 binaries when they are matched in the first clause.
> 
> I would really like to be able to send a group of binaries to the driver
> at once, as the driver is using locking a pthread mutexes directly to
> interface to some "legacy" pthread code, delivering the group of
> binaries to that "legacy" pthread (after incremeting ErlDrvBinary.refc.
> 
> The cost of sending one binary is about the same as sending 8 or 12 binaries
> at a time, due to the cost of locking and unlocking a pthread mutex, so I'd
> like to 'batch' them as much as possible, especially since the port driver
> will be receiving batches of binaries from the rest of the Erlang system.
> 
> At any rate, no matter how I implement my code, it looks like there might
> be a bug in io.c when the number of binaries exceeds 15 and their size is
> too large to compact them.
> 
> --
> Shawn.
> 
> Why do I like Erlang?  Because ``it'll drop your data for you, without
> asking you.''  (See above email.  :-)
> 
> Why do I like Perl?  Because ``in accordance with Unix tradition Perl
> gives you enough rope to hang yourself with.''
> 
> Why do I dislike Java? Because ``the class ROPE that should contain the
> method HANG to do the hanging doesn't exist because there is too much
> 'security' built into the base language.''