[erlang-questions] Port Driver, outputv, binaries.

[Tony Rogvall]

Hi!

Yes, the behavior is a bit strange.
But there a number of parameters the control how binaries and io vectors are constructed.
There are 3 types of binary representations internally:

- Heap binaries 
	Binary data is on the process heap.
	This type of binary is constructed when data is less than 64 bytes (may be changed!)

- Proc binaries
	Binary data is on the global heap. ProcBins are reference counted and  are
	mark and sweep kind of things.

- Sub binaries
	Sub binaries refer to either a  part of a ProcBin or a part of a SubBin.

When the compiler compiles binaries they may end up in different flavors, depending on
size and if they are constant or not.

When the shell constructs binaries the result may sometimes be a ProcBin when you really
expected a HeapBin to show up. In your case you really wanted a ProcBin but I thought that
a HeapBin should have been there ;-) The shell actually constructed a SubBin pointing to a
ProcBin maybe some one at the OTP team knows why!?

Next when the runtime system constructs the io vector it will preserve ProcBins and put
them in the vector for zero copy mode. HeapBins are copied in the same way as characters lists.
Everything that is not ProcBin is (internally) copied in to a new binary. Then the io vector entry will point to 
parts of that binary. Note that vectors are only broken up if the runtime hits a ProcBin.

Example:

	[1,2,3,<<100,101,102>>,[5,6,7],<<ProcBin1>>, 8,9,10,<<ProcBin2>>]

Will generate create one area with 
	NewBin = <<1,2,3,100,101,102,5,6,7,8,9,10>>
and the vector will look something like this (sketchy):

	iov[0].base = 0
	iov[0].len = 0
	binv[0] = 0
	( Index 0 is reserved packet byte stuff, header injection)

	iov[1].base = &NewBin
	iov[1].len    = 9
	binv[1] = NewBin

	iov[2].base = &ProcBin1
	iov[2].len    = size(ProcBin1)
	binv[2] = ProcBin1;

	iov[3].base = &NewBin + 9
	iov[3].len = 3
	binv[3] = NewBin

	iov[4].base = &ProcBin2
	iov[4].len    = size(ProcBin2)
	binv[4] = ProcBin2

To summarize. You can not assume how the vector will look like. The idea is that the
runtime system will try to "help" you do zero copy on "large" binaries.

/Tony






On 9 dec 2009, at 19.58, Matthew Sackman wrote:

> Hi,
> 
> So I'm writing a port driver for a linked in binary. I've implemented
> outputv, and that seems to be working fine.
> 
> In order to see what I'm being sent, I have the following in C:
> 
> void dump_ev(ErlIOVec *ev) {
>  printf("total size: %d\r\nvec len: %d\r\n", ev->size, ev->vsize);
>  int idx;
>  for (idx = 0; idx < ev->vsize; ++idx) {
>    printf("iov[%d] = ", idx);
>    SysIOVec iov = ev->iov[idx];
>    printf("[base = %p, len = %zd]\r\n", iov.iov_base, iov.iov_len);
>    printf("binv[%d] = ", idx);
>    if (NULL == ev->binv[idx]) {
>      printf("NULL\r\n");
>    } else {
>      ErlDrvBinary* bin = ev->binv[idx];
>      printf("[orig_bytes = %p; orig_size = %zd]\r\n", bin->orig_bytes, bin->orig_size);
>    }
>  }
>  printf("done\r\n");
> }
> 
> which is called at the start of outputv:
> 
> static void test_outputv(ErlDrvData drv_data, ErlIOVec *ev)
> {
>  dump_ev(ev);
>  ...
> }
> 
> Now, in the shell, I do the following:
> 
> 1> ok = erl_ddll:load_driver("ebin", "libtest"), Port = open_port({spawn_driver, libtest}, [binary]).
> 2> port_command(Port, [<<2>>,<<0,0,0,0,0,0,0,0>>,<<0>>,<<0>>,<<0>>]).
> 
> And I get out the following:
> 
> total size: 12
> vec len: 6
> iov[0] = [base = (nil), len = 0]
> binv[0] = NULL
> iov[1] = [base = 0x17a89b0, len = 1]
> binv[1] = [orig_bytes = 0x17a89b0; orig_size = 1]
> iov[2] = [base = 0x17a89e8, len = 8]
> binv[2] = [orig_bytes = 0x17a89e8; orig_size = 8]
> iov[3] = [base = 0x17a8a20, len = 1]
> binv[3] = [orig_bytes = 0x17a8a20; orig_size = 1]
> iov[4] = [base = 0x17a8a58, len = 1]
> binv[4] = [orig_bytes = 0x17a8a58; orig_size = 1]
> iov[5] = [base = 0x17a8a90, len = 1]
> binv[5] = [orig_bytes = 0x17a8a90; orig_size = 1]
> done
> 
> Which makes a lot of sense - each arg corresponds to an entry in the iov
> and binv. That's all fine.
> 
> Now, in some erlang code (a gen_server) I have this:
> 
> init([]) ->
>    erl_ddll:start(),
>    ok = erl_ddll:load_driver("ebin", ?LIBNAME),
>    Port = open_port({spawn_driver, ?LIBNAME}, [binary, stream]),
>    {ok, Port}.
> 
> handle_call({tune, BNum, APow, FPow, Opts}, _From, Port) ->
>    Data = [<<2/native>>,
>            <<BNum:64/signed-integer-native>>,
>            <<APow:8/signed-integer-native>>,
>            <<FPow:8/signed-integer-native>>,
>            <<Opts:8/native>>],
>    port_command(Port, Data),
>    ...
> 
> Then in the shell I do this:
> 
> 1> {ok, Pid} = test_drv:start_link(), gen_server:call(Pid, {tune, 0, 0, 0, 0}).
> 
> and I get out:
> 
> total size: 12
> vec len: 2
> iov[0] = [base = (nil), len = 0]
> binv[0] = NULL
> iov[1] = [base = 0x1faded0, len = 12]
> binv[1] = [orig_bytes = 0x1faded0; orig_size = 12]
> done
> 
> Why on earth has it gone and pushed all those binaries into one big
> binary? What on earth is the actual behaviour meant to be? Where is it
> defined? How can I reliably pull args out of the iov and binv? Certainly
> the behaviour is different to that in
> http://www.erlang.org/pipermail/erlang-questions/2006-March/019818.html
> 
> In short, WTF?!
> 
> Matthew
> 
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>