[erlang-questions] gen_tcp/inet/flow control/misc

[Matthias Lang]

On Friday, November 14, Florian Zumbiehl wrote:

> 1. I'm not quite sure how write-side flow control on tcp sockets works.
...
>    At the read side,

Your description of the read side agrees with what I know. I have nothing
to add.

>    At the write side, though, it's rather unclear to me how flow control
>    works. Given that there is a send_timeout option and that gen_tcp:send/2
>    can return {error,timeout}, it seems likely that gen_tcp:send/2 works
>    at least somewhat synchronously. 

With default options, everything's straightforward: gen_tcp:send/2
blocks once the TCP window is full. Easy to verify by just trying it.

If you use send_timeout, things get a bit weird and, I agree, the
documentation doesn't really tell you what to expect. IIRC, it used to
be weirder and undocumented, so things are improving.

Knowing what the underlying C interface to sockets looks like, I
expected gen_tcp:send/2 to tell me how much of the data in the call
actually got sent.  But no, gen_tcp:send/2 just returns {error,
timeout}. Actually trying it with R12B, I saw that everything gets
buffered inside Erlang and sent when the remote end starts empting the
window again. That's a bit surprising, but actually nice and reasonable.

An example:

28> f(), {ok, S} = gen_tcp:connect(localhost, 2222, [{send_timeout, 1000}]).
{ok,#Port<0.113>}
29> Bin = list_to_binary(lists:duplicate(200000, 65)).                      
<<"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"...>>
30> gen_tcp:send(S, Bin).                                                   
ok
31> gen_tcp:send(S, Bin).
ok
32> gen_tcp:send(S, Bin).
{error,timeout}
33> gen_tcp:send(S, "the end").   
{error,timeout}
34> gen_tcp:send(S, "is nigh").
ok

After command 32, the window was full. Same for 33. I then turned on
the receiver again and ran 34. The receiver got all the data from
30--34. Erlang must be buffering it. I'm not curious enough to know
where exactly.

>    On the other hand, there is the delay_send option, the
>    documentation for which claims that "the driver will use any
>    means available to queue up the message"--which sounds to me like
>    completely asynchronous operation, so that no back-pressure would
>    be felt by the inet:send/2-ing process, no matter how slow the
>    data sink. However, it's neither stated that these options were
>    somehow incompatible, nor can I find any explanation as to what
>    interaction between these options to expect.

You can probably just ignore delay_send, I don't think it's relevant
to what I think you're trying to do.

>    So, the question basically condenses down to: What buffers are
>    there in between the fd and the gen_tcp API, once again, and
>    how do they interact with the various options? 

I think there's only really one. If you don't use send_delay, it's
conceptually not there. If you do, it gets as large as it needs to be
and empties when it can.

> And how would I implement flow control at the write side?

Avoid the problem: use a protocol with flow control.

The engineer's solution: use {send_timeout, N} and resign yourself to
   not being able to know when a blocked socket becomes unblocked, at least 
   not without sending it more data.

The "pure Erlang" solution: don't use {send_timeout, N}. Instead, use
   multiple (Erlang) processes to keep track of whether the socket is 
   blocked or not.

Matt

(I know, you asked more questions. Maybe someone else can pick up 
where I left off, otherwise, maybe next week. Or read the source and
experiment.)