OTP socket.erl, unexpected interaction when receiving from errqueue

[Raimo Niskanen]

On Wed, Nov 25, 2020 at 01:42:49PM +0100, Andreas Schultz wrote:
> Am Mi., 25. Nov. 2020 um 11:37 Uhr schrieb Raimo Niskanen <
> raimo+erlang-questions@REDACTED>:
> 
> > Is it so that recvmsg(fd, &msg, MSG_ERRQUEUE) only receives from
> > the error queue, and never any regular data?
> >
> 
> That is my understanding from the man page. Experiments also confirm this.

The man page is a not entirely unambigous.  But I found a stackoverflow
thread that also confirms this (also referring to experiments).

    https://stackoverflow.com/questions/17326913/linux-udp-socket-recvmsg-with-msg-errqueue

> 
> Reading the errqueue is actually quite hard to test. The behavior for local
> errors and remote errors (e.g. reception of ICMP errors) is sometimes
> different.
> Small sample:
> 
> 1> {ok, Socket} = socket:open(inet, dgram, udp).
> {ok,{'$socket',#Ref<0.1577644963.140640257.162512>}}
> 2> ok = socket:setopt(Socket, ip, recverr, true).
> ok
> 3> Dest = #{family => inet, addr => {127,0,0,1}, port => 1234}.
> #{addr => {127,0,0,1},family => inet,port => 1234}
> 4> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> ok
> 5> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> {error,{econnrefused,[<<"Data">>]}}

Weird to get econnrefused from sendto() on an unconnected dgram socket.

> 
> 
> The first sendto returns `ok`, the error can be read in subsequent recvmsg.
> The second sendto returns the error immediately because the kernel has
> learned that the local endpoint does not exist.
> 
> 
> 
> > Today there is no VM support for a NIF do differ between POLLIN
> > and POLLERR.  I have asked the VM guys to have a look at that.
> >
> > Without that you can as your response to receiving
> > {'$socket',Socket,select,SelectHandle}
> > call socket:recvmsg(Socket, [errqueue], 0), to poll, and then
> > if the poll gave {error,timeout} call socket:recvmsg(Socket, 0, nowait).
> >
> 
> If was actually thinking about doing `socket:recvmsg(Socket, [errqueue],
> nowait)`
> if
>    a) just received a select message and
>    b) socket:recvfrom returned a new select info instead of reading any data
> 
> That should capture the situation where only data in the errqueue is
> present without having to
> use the 0 timeout.

Might there not be a possibility to starve out error messages in the face of
continously incoming data with this approach?

> 
> Then we will have to optimize Timeout =:= 0, and maybe introduce
> > Timeout =:= poll with a nicer return value for no data.
> >
> 
> I like the general idea, but the `nowait` option now feels a  bit wrong. We
> end up with `nowait == do a select` and `poll == just check if there is
> data`.
> It might be too late for changing `nowait`, but what about adding `select`
> as an alias to `nowait` ?

I think it is too late to change 'nowait' now, and it is not entirely off.
It states that the call will not wait, but not what happens instead.
While 'select' states what to do, but only sometimes.

'poll' also states what to do.  We could add 'select' as an alias for
'nowait', some might find it more appropriate.

We thought about having only polling operations and a dedicated select
operation, but that would need minimum two NIF calls also when handling
much data, so we decided on doing automatic select depending on the
result of the operation.

/ Raimo


> 
> Regards,
> Andreas
> 
> Timeout =:= 0 today causes quite some overhead, but it should work.
> >
> > Cheers
> > / Raimo
> >
> >
> > On Mon, Nov 23, 2020 at 05:42:23PM +0100, Andreas Schultz wrote:
> > > Hi,
> > >
> > > If setup a socket with:
> > >
> > >     socket:setopt(Socket, ip, recverr, true)
> > >
> > > If then started a asynchronous recvmsg with:
> > >
> > >     {select, SelectInfo} = socket:recvfrom(Socket, 0, [], nowait)
> > >
> > > When now something arrives in the error queue, I'll get a select info
> > > message with:
> > >
> > >     {'$socket', Socket, select, SelectInfo}
> > >
> > > The problem is, nothing in there tells me to read from the error queue.
> > The
> > > underlying OS poll/epoll call would have this information, but it is lost
> > > in the Erlang message.
> > >
> > > When I now try to read from the socket with:
> > >
> > >    socket:recvfrom(Socket, 0, [], nowait)
> > >
> > > all I get is another `{select, SelectInfo}` tuple, followed by another
> > > `{'$socket', Socket, select, SelectInfo}` messages.
> > > This can actually end up in an endless busy loop.
> > >
> > > To actually clear this I would to do a:
> > >
> > >     socket:recvmsg(Socket, [errqueue], nowait)
> > >
> > > On an POSIX socket, I would have to actually poll for POLLIN | POLLERR to
> > > get a similar behavior. But the return of the poll would tell me whether
> > it
> > > was POLLIN or POLLERR (similar for epoll).
> > > For the Erlang API it might sense to always poll for both conditions, but
> > > we then should get an indication what exactly it was.
> > >
> > > Would it be possible to change the $socket message to something like:
> > >
> > >     {'$socket', Socket, error, SelectInfo}
> > >
> > > for POLLERR/EPOLLERR ???
> > >
> > > Regards
> > > Andreas
> > > --
> > >
> > > Andreas Schultz
> >
> > --
> >
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> >
> 
> 
> -- 
> 
> Andreas Schultz

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB