[erlang-questions] Troubling gen_tcp.send/3 performance

[Fred Hebert]

On 12/04, Matthew Shapiro wrote:
>Yeah I think I can safely say this is something to do with how VLC is
>functioning.  Even remotely I'm getting stuck in send, though not as bad,
>and looking at what's going on in the difference between VLC and Ffmpeg I
>believe my timestamps are being unexpected and both are reacting weirdly
>too it, with VLC trying to slow the feed down for whatever reason and that
>may be causing it to be slow to do a TCP read.  Shot in the dark but at
>least I know now that it's not an BEAM issue (and my time on this hasn't
>been to waste).
>
>So sorry for the thread when Erlang/Beam isn't at fault in the end :)
>

It is and it is not. One of the interesting things with the TCP stack is 
that it sends a signal through ports, and if the send queue is full, the 
whole thing blocks; it however blocks with live data in there, and 
there's no good way to give it up.

The reason is blocks that the TCP work is done over a port program, 
meaning message passing takes place:

    Erlang Process               TCP port program
         |                               |
         |----------- Data ------------->|
         |                               |
         |<--- {inet_reply,S,Status} ----|

This is roughly what takes place within the thing. This is done through 
calling erlang:port_command/2-3. The thing though is that if the buffer 
of the port program is full, you get descheduled until there's place.

There's a cheat though, one we at Heroku have used in Logplex. The trick 
is to bypass the stack in critical 'must never block' circumstances and 
call port_command yourself, but by passing in the option list 
'[nosuspend]'.

When you do this and the port program representing the TCP socket is 
busy, you'll get a 'false' return value indicating it failed because it 
was full (you get 'true' when it works):

    case erlang:port_command(Socket, Data, [nosuspend]) of
        false ->
            handle_busy(...);
        true ->
            receive
                {inet_reply, Socket, Status} -> % Status == send return
                    Status
            end
    end

This will let you do things like figure out the consumer side (in this 
case VLC) is too slow, and possibly give up sending a few frames and 
send newer ones later.

Control flow is fun!

Regards,
Fred.