OTP socket.erl, unexpected interaction when receiving from errqueue

[Raimo Niskanen]

The VM guys ran into problems when testing this feature.
It seems to be not very portable, e.g on platforms
with only select there is no notion of an error condition,
neither for epoll.  It is only poll that can supply this info,
and what renders a POLLERR differs between platforms.

Furtermore I have done some manual testing of the socket
option {ip,recverr}:

Erlang/OTP 24 [DEVELOPMENT] [erts-11.1.3] [source-aaa3fc53a3] [64-bit]
[smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Eshell V11.1.3  (abort with ^G)
1> {ok, S} = socket:open(inet, dgram).
      
{ok,{'$socket',#Ref<0.204328729.875954179.14364>}}
2> socket:setopt(S, {ip,recverr}, true).
      
ok
3> socket:recvmsg(S, [], nowait).
        
{select,{select_info,recvmsg,
                     #Ref<0.204328729.875823107.14380>}}
4> flush().                             
ok
5> socket:sendto(S, "hello", #{family => inet, addr => {127,0,0,1}, port =>
44444}). 
ok
6> 
6> flush().
        
Shell got {'$socket',{'$socket',#Ref<0.204328729.875954179.14364>},
                     select,#Ref<0.204328729.875823107.14380>}
ok
7> socket:recvmsg(S, [], nowait).
        
{error,econnrefused}
8> socket:recvmsg(S, [errqueue], 0).     
{ok,#{addr =>
          #{addr => {127,0,0,1},family => inet,port => 44444},
      ctrl =>
          [#{data =>

<<111,0,0,0,2,3,3,0,0,0,0,0,0,0,0,0,2,0,0,0,127,0,0,1,...>>,
             level => ip,type => recverr,
             value =>
                 #{code => port_unreach,data => 0,error => econnrefused,
                   info => 0,
                   offender => #{addr => {127,0,0,1},family => inet,port =>
0},
                   origin => icmp,type => dest_unreach}}],
      flags => [errqueue],
      iov => [<<"hello">>]}}
9> socket:recvmsg(S, [errqueue], 0).
{error,timeout}
10> socket:recvmsg(S, [], nowait).   
{select,{select_info,recvmsg,
                     #Ref<0.204328729.875823107.14427>}}
11> flush().                         
ok


We get a select message, do a normal read.  It returns an error,
so we read the error queue to get a verbose error message.

I think this approach might be a good one.  When getting an error; 
read the error queue with timeout 0, maybe twice to give it higher
effective priority, then back to normal reading.

What do you think?

/ Raimo


On Fri, Dec 04, 2020 at 10:45:03AM +0100, Raimo Niskanen wrote:
> The VM team has now planned a task to extend the NIF API to handle
> select/poll on error i.e enif_select_error() and ERL_NIF_SELECT_ERROR.
> 
> When that is done we can use that in the socket API, probably exactly as
> you first suggested.
> 
> A question is if we should just add that, and all code using
> the socket API has to be prepared for a new message, or if we need
> an option on the 'otp' protocol level.
> 
> The Experimental status of the socket API allows for such a
> backwards-incompatible change, but that does not mean that
> we need to do one...
> 
> Until then I have just merged, into 'master', an optimization of
> recv with Timeout =:= 0 so it skips the select/cancel select dance.
> 
> Cheers
> / Raimo
> 
> 
> On Wed, Nov 25, 2020 at 02:45:22PM +0100, Raimo Niskanen wrote:
> > On Wed, Nov 25, 2020 at 01:42:49PM +0100, Andreas Schultz wrote:
> > > Am Mi., 25. Nov. 2020 um 11:37 Uhr schrieb Raimo Niskanen <
> > > raimo+erlang-questions@REDACTED>:
> > > 
> > > > Is it so that recvmsg(fd, &msg, MSG_ERRQUEUE) only receives from
> > > > the error queue, and never any regular data?
> > > >
> > > 
> > > That is my understanding from the man page. Experiments also confirm this.
> > 
> > The man page is a not entirely unambigous.  But I found a stackoverflow
> > thread that also confirms this (also referring to experiments).
> > 
> >     https://stackoverflow.com/questions/17326913/linux-udp-socket-recvmsg-with-msg-errqueue
> > 
> > > 
> > > Reading the errqueue is actually quite hard to test. The behavior for local
> > > errors and remote errors (e.g. reception of ICMP errors) is sometimes
> > > different.
> > > Small sample:
> > > 
> > > 1> {ok, Socket} = socket:open(inet, dgram, udp).
> > > {ok,{'$socket',#Ref<0.1577644963.140640257.162512>}}
> > > 2> ok = socket:setopt(Socket, ip, recverr, true).
> > > ok
> > > 3> Dest = #{family => inet, addr => {127,0,0,1}, port => 1234}.
> > > #{addr => {127,0,0,1},family => inet,port => 1234}
> > > 4> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> > > ok
> > > 5> socket:sendto(Socket, <<"Data">>, Dest, nowait).
> > > {error,{econnrefused,[<<"Data">>]}}
> > 
> > Weird to get econnrefused from sendto() on an unconnected dgram socket.
> > 
> > > 
> > > 
> > > The first sendto returns `ok`, the error can be read in subsequent recvmsg.
> > > The second sendto returns the error immediately because the kernel has
> > > learned that the local endpoint does not exist.
> > > 
> > > 
> > > 
> > > > Today there is no VM support for a NIF do differ between POLLIN
> > > > and POLLERR.  I have asked the VM guys to have a look at that.
> > > >
> > > > Without that you can as your response to receiving
> > > > {'$socket',Socket,select,SelectHandle}
> > > > call socket:recvmsg(Socket, [errqueue], 0), to poll, and then
> > > > if the poll gave {error,timeout} call socket:recvmsg(Socket, 0, nowait).
> > > >
> > > 
> > > If was actually thinking about doing `socket:recvmsg(Socket, [errqueue],
> > > nowait)`
> > > if
> > >    a) just received a select message and
> > >    b) socket:recvfrom returned a new select info instead of reading any data
> > > 
> > > That should capture the situation where only data in the errqueue is
> > > present without having to
> > > use the 0 timeout.
> > 
> > Might there not be a possibility to starve out error messages in the face of
> > continously incoming data with this approach?
> > 
> > > 
> > > Then we will have to optimize Timeout =:= 0, and maybe introduce
> > > > Timeout =:= poll with a nicer return value for no data.
> > > >
> > > 
> > > I like the general idea, but the `nowait` option now feels a  bit wrong. We
> > > end up with `nowait == do a select` and `poll == just check if there is
> > > data`.
> > > It might be too late for changing `nowait`, but what about adding `select`
> > > as an alias to `nowait` ?
> > 
> > I think it is too late to change 'nowait' now, and it is not entirely off.
> > It states that the call will not wait, but not what happens instead.
> > While 'select' states what to do, but only sometimes.
> > 
> > 'poll' also states what to do.  We could add 'select' as an alias for
> > 'nowait', some might find it more appropriate.
> > 
> > We thought about having only polling operations and a dedicated select
> > operation, but that would need minimum two NIF calls also when handling
> > much data, so we decided on doing automatic select depending on the
> > result of the operation.
> > 
> > / Raimo
> > 
> > 
> > > 
> > > Regards,
> > > Andreas
> > > 
> > > Timeout =:= 0 today causes quite some overhead, but it should work.
> > > >
> > > > Cheers
> > > > / Raimo
> > > >
> > > >
> > > > On Mon, Nov 23, 2020 at 05:42:23PM +0100, Andreas Schultz wrote:
> > > > > Hi,
> > > > >
> > > > > If setup a socket with:
> > > > >
> > > > >     socket:setopt(Socket, ip, recverr, true)
> > > > >
> > > > > If then started a asynchronous recvmsg with:
> > > > >
> > > > >     {select, SelectInfo} = socket:recvfrom(Socket, 0, [], nowait)
> > > > >
> > > > > When now something arrives in the error queue, I'll get a select info
> > > > > message with:
> > > > >
> > > > >     {'$socket', Socket, select, SelectInfo}
> > > > >
> > > > > The problem is, nothing in there tells me to read from the error queue.
> > > > The
> > > > > underlying OS poll/epoll call would have this information, but it is lost
> > > > > in the Erlang message.
> > > > >
> > > > > When I now try to read from the socket with:
> > > > >
> > > > >    socket:recvfrom(Socket, 0, [], nowait)
> > > > >
> > > > > all I get is another `{select, SelectInfo}` tuple, followed by another
> > > > > `{'$socket', Socket, select, SelectInfo}` messages.
> > > > > This can actually end up in an endless busy loop.
> > > > >
> > > > > To actually clear this I would to do a:
> > > > >
> > > > >     socket:recvmsg(Socket, [errqueue], nowait)
> > > > >
> > > > > On an POSIX socket, I would have to actually poll for POLLIN | POLLERR to
> > > > > get a similar behavior. But the return of the poll would tell me whether
> > > > it
> > > > > was POLLIN or POLLERR (similar for epoll).
> > > > > For the Erlang API it might sense to always poll for both conditions, but
> > > > > we then should get an indication what exactly it was.
> > > > >
> > > > > Would it be possible to change the $socket message to something like:
> > > > >
> > > > >     {'$socket', Socket, error, SelectInfo}
> > > > >
> > > > > for POLLERR/EPOLLERR ???
> > > > >
> > > > > Regards
> > > > > Andreas
> > > > > --
> > > > >
> > > > > Andreas Schultz
> > > >
> > > > --
> > > >
> > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > > >
> > > 
> > > 
> > > -- 
> > > 
> > > Andreas Schultz
> > 
> > -- 
> > 
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> 
> -- 
> 
> / Raimo Niskanen, Erlang/OTP, Ericsson AB

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB