ports, binaries & iovecs

[Scott Lystig Fritchie]

>>>>> "vs" == Vance Shipley <vances@REDACTED> writes:

vs> I am finding that sending binaries to a port with port_command/2,
vs> and receiving them with outputv(), doesn't work the way I had
vs> expected.

I spent a bunch of time trying to figure that out a couple of months
ago.  Eventually I figured it out ... though the answer here may not
be correct.  {scratches head and re-examines io.c}

The magic is done by the IO_LIST_VEC_COUNT macro in io.c, which is
called by the function io_list_vec_len().

io_list_vec_len() will take an I/O list with a mix of bytes and
binaries and put all the individual bytes into a single contiguous
buffer allocated for that purpose.  The ErlIOVec and SysIOVec
structures keep the info to keep everything in the correct order,
naturally.

Binaries are weird because there are several types.  If a binary is a
reference counted thing (its subtag == REFC_BINARY_SUBTAG), and if its
size is greater than ERL_SMALL_IO_BIN_LIMIT (which is
(4*ERL_ONHEAP_BIN_LIMIT), which is 256), then it gets its own iovec
entry.  If the binary is a refcounted binary but smaller than that, or
if it's a heap binary (which, by definition, is small because it's
stored entirely on the Erlang process's heap), then the data will be
copied into the buffer used by individual bytes.

Within some range of error, that's what io_list_vec_len() calculates.
The Real Truth may be slightly different.  What io_list_to_vec() does
is a slightly different story.  (My interpretation may be incorrect
here, too.)

If io_list_vec_len() figures the entire iovec is <= SMALL_WRITE_VEC
(16), then it sets its local variable "blimit" to 0 and passes
"blimit" as the last arg to io_list_to_vec().  Once again, the type of
binary comes into play:

	if (binary is reference counted) {
		if (bin_limit && binary's size < bin_limit) {
			copy into individual bytes' buffer
		} else {
			/* big binary _or_ we don't have iovec size worries */
			keep it a separate vector item
		}
	} else {
		/* We're a heap binary if we're here, thus we're small */
		copy into individual bytes' buffer
	}

According to erl_binary.h and binary.c, the maximum size of an on-heap
binary is governed by:

	#define ERL_ONHEAP_BIN_LIMIT 64

-Scott