[erlang-bugs] gen_tcp:send/2 gets stuck despite send_timeout

[Steve Vinoski]

On Tue, Oct 28, 2014 at 8:46 AM, Holger Weiß <holger@REDACTED>
wrote:

> Hi there,
>
> I'm an ejabberd contributor, and we're currently facing the issue that
> gen_tcp:send/2 occasionally blocks forever even though a 'send_timeout'
> (and 'send_timeout_close') has been specified.¹  This seems to happen
> only under rare circumstances, but when it happens, it can crash the VM,
> as the process that's stuck in the gen_tcp:send/2 call stops processing
> its message queue and therefore eats the available memory, eventually.
>
> This *only* seems to happen when epoll(7) is used, i.e. when "+K true"
> is specified on Linux.  "+K false" makes the issue go away.
>
> Also, it only happens when the TCP socket is no longer usable.  In the
> past, it could occur that an ejabberd process called gen_tcp:send/2 even
> though an earlier call returned a failure already.  Since we changed the
> code to fix that, the issue is triggered less frequently; and in those
> cases where it still *is* triggered, it's obvious from looking at the
> details that the socket got closed more or less at the same time.
>
> The problem is that I'm not able to reproduce this myself.  So far,
> we've only been made aware of this issue on two servers, both of them
> running in production, and it's only easily reproducible on one of them.
> That one is running Erlang 17.1 on a Xen instance (I guess I could ask
> the admin to update to 17.3).
>
> Without code to reproduce the issue, this is probably non-trivial to
> debug :-(  At least there's one live system where the issue is usually
> triggered multiple times per day.  Any suggestions on how to proceed?
>
> Thanks, Holger
>
> ¹ According to process_info/1, the current function is prim_inet:send/3.
>

I've seen this happen when the receiver is not reading from its socket,
thus causing the TCP window to close to apply backpressure to the sender,
and so Erlang's inet driver fills the kernel's send buffers and then
inet_drv fills its own buffers with data to be sent. Once those are all
full, and you try to send again, prim_inet:send blocks waiting to receive a
message from inet_drv that won't be sent until some send buffer space is
freed up due to the receiver reading data from its socket. Under these
conditions, netstat will show your connection's kernel buffers to be full
and unchanging, and a call to erlang:port_info(Socket, queue_size) will
show queued data greater than the default of 8k for that socket. It can sit
that way forever until the receiver reads.

One thing that might help under these conditions is to send data directly
via port_command like this:

    erlang:port_command(Socket, Data, [nosuspend])

which returns false if the port/socket is busy, but even this doesn't seem
to be foolproof, nor does relying on send_timeout. I've seen both work, and
in fact both worked as expected in little cases I tried before sending this
reply, but I've also seen cases where I expected them to work but they
didn't and I couldn't determine why.

--steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20141028/f8cb8a77/attachment.htm>