<div dir="ltr"><div dir="ltr">Am Fr., 8. Jan. 2021 um 16:44 Uhr schrieb Raimo Niskanen <<a href="mailto:raimo%2Berlang-questions@erlang.org" target="_blank">raimo+erlang-questions@erlang.org</a>>:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The VM guys ran into problems when testing this feature.<br>
It seems to be not very portable, e.g on platforms<br>
with only select there is no notion of an error condition,<br>
neither for epoll. It is only poll that can supply this info,<br>
and what renders a POLLERR differs between platforms.<br></blockquote><div><br></div><div>IP_RECVERR is Linux specific, it is therefore not surprising that other OSes behave differently.</div><div>From the Linux man page, I would expect that EPOLLERR works,<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Furtermore I have done some manual testing of the socket<br>
option {ip,recverr}:<br>
<br>
Erlang/OTP 24 [DEVELOPMENT] [erts-11.1.3] [source-aaa3fc53a3] [64-bit]<br>
[smp:8:8] [ds:8:8:10] [async-threads:1] [jit]<br>
<br>
Eshell V11.1.3 (abort with ^G)<br>
1> {ok, S} = socket:open(inet, dgram).<br>
<br>
{ok,{'$socket',#Ref<0.204328729.875954179.14364>}}<br>
2> socket:setopt(S, {ip,recverr}, true).<br>
<br>
ok<br>
3> socket:recvmsg(S, [], nowait).<br>
<br>
{select,{select_info,recvmsg,<br>
#Ref<0.204328729.875823107.14380>}}<br>
4> flush(). <br>
ok<br>
5> socket:sendto(S, "hello", #{family => inet, addr => {127,0,0,1}, port =><br>
44444}).</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
ok<br>
6> <br>
6> flush().<br>
<br>
Shell got {'$socket',{'$socket',#Ref<0.204328729.875954179.14364>},<br>
select,#Ref<0.204328729.875823107.14380>}<br>
ok<br>
7> socket:recvmsg(S, [], nowait).<br>
<br>
{error,econnrefused}<br>
8> socket:recvmsg(S, [errqueue], 0). <br>
{ok,#{addr =><br>
#{addr => {127,0,0,1},family => inet,port => 44444},<br>
ctrl =><br>
[#{data =><br>
<br>
<<111,0,0,0,2,3,3,0,0,0,0,0,0,0,0,0,2,0,0,0,127,0,0,1,...>>,<br>
level => ip,type => recverr,<br>
value =><br>
#{code => port_unreach,data => 0,error => econnrefused,<br>
info => 0,<br>
offender => #{addr => {127,0,0,1},family => inet,port =><br>
0},<br>
origin => icmp,type => dest_unreach}}],<br>
flags => [errqueue],<br>
iov => [<<"hello">>]}}<br>
9> socket:recvmsg(S, [errqueue], 0).<br>
{error,timeout}<br>
10> socket:recvmsg(S, [], nowait). <br>
{select,{select_info,recvmsg,<br>
#Ref<0.204328729.875823107.14427>}}<br>
11> flush(). <br>
ok<br>
<br>
<br>
We get a select message, do a normal read. It returns an error,<br>
so we read the error queue to get a verbose error message.<br></blockquote><div><br></div><div>The sequence of events very much depends on the state of the socket when you call the recvfrom/recvmsg, leading to an unstable behavior.</div><div>Consider this example:</div><div><br></div><div>Erlang/OTP 24 [DEVELOPMENT] [erts-11.1.4] [source-a348f5a237] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]<br><br>Eshell V11.1.4 (abort with ^G)<br>1> {ok, S} = socket:open(inet, dgram).<br>{ok,{'$socket',#Ref<0.1380161320.4108451842.177694>}}<br>2> socket:setopt(S, {ip,recverr}, true).<br>ok<br>3> socket:sendto(S, "hello", #{family => inet, addr => {127, 0, 0, 1}, port => 44444}). <br>ok<br>4> socket:sendto(S, "hello", #{family => inet, addr => {127, 0, 0, 1}, port => 44444}).<br>{error,econnrefused}<br>5> socket:recvfrom(S, 0, [], nowait).<br>{select,{select_info,recvfrom,<br> #Ref<0.1380161320.4108320780.177852>}}<br>6> flush().<br>Shell got {'$socket',{'$socket',#Ref<0.1380161320.4108451842.177694>},<br> select,#Ref<0.1380161320.4108320780.177852>}<br>ok<br>7> socket:recvfrom(S, 0, [], nowait).<br>{select,{select_info,recvfrom,<br> #Ref<0.1380161320.4108320780.177867>}}<br>8> flush(). <br>Shell got {'$socket',{'$socket',#Ref<0.1380161320.4108451842.177694>},<br> select,#Ref<0.1380161320.4108320780.177867>}<br></div><div><br></div><div>We get an endless stream of select_infos without any hint that there might be something hiding in the error queue.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I think this approach might be a good one. When getting an error; <br>
read the error queue with timeout 0, maybe twice to give it higher<br>
effective priority, then back to normal reading.<br></blockquote><div><br></div><div>To avoid the endless select loop from above, the user would have to do a *speculative / blind* read of the error queue.</div><div><br></div><div>The IP_RECVERR flag has to be set explicitly, such a situation can therefore not sneak up unexpectedly on a user. So this would IMHO be acceptable, but needs very explicit documentation somewhere (maybe near the recverr flag?)<br><br>I still would prefer to get the POLLERR vs. POLLIN indication in the select message. Even when it only works on Linux and not on others.</div><div>The MSG_ERRQUEUE recvmsg flag and the whole IP_RECVERR mechanism to get the payload of ICMP errors for UDP sockets is not portable and only usable on Linux anyhow.<br></div><div>The classical approach on e.g. FreeBSD seems to involve an additional RAW socket to read the ICMPs. Not sure how/if that is supposed to work on Windows or OSX.</div><div><br></div><div>The problem I see with reporting the poll flags is that they are a bitfield. Having both POLLIN and POLLERR set is therefore possible. Representing this as a map or proplist seems a bit wasteful.</div><div>Since we always only request `POLLIN | POLLERR` and `POLLOUT | POLLERR`, maybe using a select_info tuple of `{select_info, select_tag(), select_ref(), select_opt()}` with `select_opt() :: {ReadWrite :: boolean(), Error :: boolean()}` could work ???</div><div><br></div><div>Regards,</div><div>Andreas</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
What do you think?<br>
<br>
/ Raimo<br>
<br>
<br>
On Fri, Dec 04, 2020 at 10:45:03AM +0100, Raimo Niskanen wrote:<br>
> The VM team has now planned a task to extend the NIF API to handle<br>
> select/poll on error i.e enif_select_error() and ERL_NIF_SELECT_ERROR.<br>
> <br>
> When that is done we can use that in the socket API, probably exactly as<br>
> you first suggested.<br>
> <br>
> A question is if we should just add that, and all code using<br>
> the socket API has to be prepared for a new message, or if we need<br>
> an option on the 'otp' protocol level.<br>
> <br>
> The Experimental status of the socket API allows for such a<br>
> backwards-incompatible change, but that does not mean that<br>
> we need to do one...<br>
> <br>
> Until then I have just merged, into 'master', an optimization of<br>
> recv with Timeout =:= 0 so it skips the select/cancel select dance.<br>
> <br>
> Cheers<br>
> / Raimo<br>
> <br>
> <br>
> On Wed, Nov 25, 2020 at 02:45:22PM +0100, Raimo Niskanen wrote:<br>
> > On Wed, Nov 25, 2020 at 01:42:49PM +0100, Andreas Schultz wrote:<br>
> > > Am Mi., 25. Nov. 2020 um 11:37 Uhr schrieb Raimo Niskanen <<br>
> > > <a href="mailto:raimo%2Berlang-questions@erlang.org" target="_blank">raimo+erlang-questions@erlang.org</a>>:<br>
> > > <br>
> > > > Is it so that recvmsg(fd, &msg, MSG_ERRQUEUE) only receives from<br>
> > > > the error queue, and never any regular data?<br>
> > > ><br>
> > > <br>
> > > That is my understanding from the man page. Experiments also confirm this.<br>
> > <br>
> > The man page is a not entirely unambigous. But I found a stackoverflow<br>
> > thread that also confirms this (also referring to experiments).<br>
> > <br>
> > <a href="https://stackoverflow.com/questions/17326913/linux-udp-socket-recvmsg-with-msg-errqueue" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/17326913/linux-udp-socket-recvmsg-with-msg-errqueue</a><br>
> > <br>
> > > <br>
> > > Reading the errqueue is actually quite hard to test. The behavior for local<br>
> > > errors and remote errors (e.g. reception of ICMP errors) is sometimes<br>
> > > different.<br>
> > > Small sample:<br>
> > > <br>
> > > 1> {ok, Socket} = socket:open(inet, dgram, udp).<br>
> > > {ok,{'$socket',#Ref<0.1577644963.140640257.162512>}}<br>
> > > 2> ok = socket:setopt(Socket, ip, recverr, true).<br>
> > > ok<br>
> > > 3> Dest = #{family => inet, addr => {127,0,0,1}, port => 1234}.<br>
> > > #{addr => {127,0,0,1},family => inet,port => 1234}<br>
> > > 4> socket:sendto(Socket, <<"Data">>, Dest, nowait).<br>
> > > ok<br>
> > > 5> socket:sendto(Socket, <<"Data">>, Dest, nowait).<br>
> > > {error,{econnrefused,[<<"Data">>]}}<br>
> > <br>
> > Weird to get econnrefused from sendto() on an unconnected dgram socket.<br>
> > <br>
> > > <br>
> > > <br>
> > > The first sendto returns `ok`, the error can be read in subsequent recvmsg.<br>
> > > The second sendto returns the error immediately because the kernel has<br>
> > > learned that the local endpoint does not exist.<br>
> > > <br>
> > > <br>
> > > <br>
> > > > Today there is no VM support for a NIF do differ between POLLIN<br>
> > > > and POLLERR. I have asked the VM guys to have a look at that.<br>
> > > ><br>
> > > > Without that you can as your response to receiving<br>
> > > > {'$socket',Socket,select,SelectHandle}<br>
> > > > call socket:recvmsg(Socket, [errqueue], 0), to poll, and then<br>
> > > > if the poll gave {error,timeout} call socket:recvmsg(Socket, 0, nowait).<br>
> > > ><br>
> > > <br>
> > > If was actually thinking about doing `socket:recvmsg(Socket, [errqueue],<br>
> > > nowait)`<br>
> > > if<br>
> > > a) just received a select message and<br>
> > > b) socket:recvfrom returned a new select info instead of reading any data<br>
> > > <br>
> > > That should capture the situation where only data in the errqueue is<br>
> > > present without having to<br>
> > > use the 0 timeout.<br>
> > <br>
> > Might there not be a possibility to starve out error messages in the face of<br>
> > continously incoming data with this approach?<br>
> > <br>
> > > <br>
> > > Then we will have to optimize Timeout =:= 0, and maybe introduce<br>
> > > > Timeout =:= poll with a nicer return value for no data.<br>
> > > ><br>
> > > <br>
> > > I like the general idea, but the `nowait` option now feels a bit wrong. We<br>
> > > end up with `nowait == do a select` and `poll == just check if there is<br>
> > > data`.<br>
> > > It might be too late for changing `nowait`, but what about adding `select`<br>
> > > as an alias to `nowait` ?<br>
> > <br>
> > I think it is too late to change 'nowait' now, and it is not entirely off.<br>
> > It states that the call will not wait, but not what happens instead.<br>
> > While 'select' states what to do, but only sometimes.<br>
> > <br>
> > 'poll' also states what to do. We could add 'select' as an alias for<br>
> > 'nowait', some might find it more appropriate.<br>
> > <br>
> > We thought about having only polling operations and a dedicated select<br>
> > operation, but that would need minimum two NIF calls also when handling<br>
> > much data, so we decided on doing automatic select depending on the<br>
> > result of the operation.<br>
> > <br>
> > / Raimo<br>
> > <br>
> > <br>
> > > <br>
> > > Regards,<br>
> > > Andreas<br>
> > > <br>
> > > Timeout =:= 0 today causes quite some overhead, but it should work.<br>
> > > ><br>
> > > > Cheers<br>
> > > > / Raimo<br>
> > > ><br>
> > > ><br>
> > > > On Mon, Nov 23, 2020 at 05:42:23PM +0100, Andreas Schultz wrote:<br>
> > > > > Hi,<br>
> > > > ><br>
> > > > > If setup a socket with:<br>
> > > > ><br>
> > > > > socket:setopt(Socket, ip, recverr, true)<br>
> > > > ><br>
> > > > > If then started a asynchronous recvmsg with:<br>
> > > > ><br>
> > > > > {select, SelectInfo} = socket:recvfrom(Socket, 0, [], nowait)<br>
> > > > ><br>
> > > > > When now something arrives in the error queue, I'll get a select info<br>
> > > > > message with:<br>
> > > > ><br>
> > > > > {'$socket', Socket, select, SelectInfo}<br>
> > > > ><br>
> > > > > The problem is, nothing in there tells me to read from the error queue.<br>
> > > > The<br>
> > > > > underlying OS poll/epoll call would have this information, but it is lost<br>
> > > > > in the Erlang message.<br>
> > > > ><br>
> > > > > When I now try to read from the socket with:<br>
> > > > ><br>
> > > > > socket:recvfrom(Socket, 0, [], nowait)<br>
> > > > ><br>
> > > > > all I get is another `{select, SelectInfo}` tuple, followed by another<br>
> > > > > `{'$socket', Socket, select, SelectInfo}` messages.<br>
> > > > > This can actually end up in an endless busy loop.<br>
> > > > ><br>
> > > > > To actually clear this I would to do a:<br>
> > > > ><br>
> > > > > socket:recvmsg(Socket, [errqueue], nowait)<br>
> > > > ><br>
> > > > > On an POSIX socket, I would have to actually poll for POLLIN | POLLERR to<br>
> > > > > get a similar behavior. But the return of the poll would tell me whether<br>
> > > > it<br>
> > > > > was POLLIN or POLLERR (similar for epoll).<br>
> > > > > For the Erlang API it might sense to always poll for both conditions, but<br>
> > > > > we then should get an indication what exactly it was.<br>
> > > > ><br>
> > > > > Would it be possible to change the $socket message to something like:<br>
> > > > ><br>
> > > > > {'$socket', Socket, error, SelectInfo}<br>
> > > > ><br>
> > > > > for POLLERR/EPOLLERR ???<br>
> > > > ><br>
> > > > > Regards<br>
> > > > > Andreas<br>
> > > > > --<br>
> > > > ><br>
> > > > > Andreas Schultz<br>
> > > ><br>
> > > > --<br>
> > > ><br>
> > > > / Raimo Niskanen, Erlang/OTP, Ericsson AB<br>
> > > ><br>
> > > <br>
> > > <br>
> > > -- <br>
> > > <br>
> > > Andreas Schultz<br>
> > <br>
> > -- <br>
> > <br>
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB<br>
> <br>
> -- <br>
> <br>
> / Raimo Niskanen, Erlang/OTP, Ericsson AB<br>
<br>
-- <br>
<br>
/ Raimo Niskanen, Erlang/OTP, Ericsson AB<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><p><span style="font-family:verdana,geneva,sans-serif;font-size:10pt">Andreas Schultz</span></p>
<p><span style="font-family:verdana,geneva,sans-serif;font-size:10pt">-- </span></p>
<p><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">Principal Engineer</span></p>
<p><span style="font-family:verdana,geneva,sans-serif;font-size:10.6667px"> t: +49 391 819099-224 </span></p>
<p><span style="font-family:verdana,geneva,sans-serif;font-size:8pt"><span style="font-family:verdana,geneva,sans-serif;font-size:10.6667px"></span></span></p>
<p><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">------------------------------- enabling your networks -----------------------------</span></p>
<table style="height:142px;width:550px">
<tbody>
<tr style="height:30px">
<td style="width:146px;height:30px">
<p><span style="font-family:verdana,geneva,sans-serif;color:rgb(34,34,34);font-size:8pt">Travelping GmbH <br></span><span style="font-family:verdana,geneva,sans-serif;color:rgb(34,34,34);font-size:8pt">Roentgenstraße 13<br></span><span style="font-family:verdana,geneva,sans-serif;color:rgb(34,34,34);font-size:8pt">39108 Magdeburg<br></span><span style="font-family:verdana,geneva,sans-serif;color:rgb(34,34,34);font-size:8pt">Germany<br><br></span></p>
</td>
<td style="width:162px;height:30px">
<p style="color:rgb(34,34,34);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:small;text-align:justify"><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">t: +49 391 819099-0<br></span><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">f: +49 391 819099-299</span></p>
<p style="color:rgb(34,34,34);font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:small;text-align:justify"><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">e: <a href="mailto:info@travelping.com" target="_blank">info@travelping.com</a><br></span><span style="font-family:verdana,geneva,sans-serif;font-size:8pt">w: <a href="https://www.travelping.com/" target="_blank">https://www.travelping.com/</a></span></p>
</td>
</tr>
<tr style="height:17px">
<td style="width:146px;height:17px"><span style="font-size:8pt;font-family:verdana,geneva,sans-serif">Company registration: Amtsgericht Stendal <br>Managing Director: Holger Winkelmann<br></span></td>
<td style="width:162px;height:17px"><span style="font-size:8pt;font-family:verdana,geneva,sans-serif">Reg. No.: HRB 10578<br>VAT ID: DE236673780</span></td>
</tr>
</tbody>
</table></div></div>