<div dir="ltr">Hi Oliver,<div><br></div><div>One reason for you problems might be the fact that Erlangs SCTP implementation is unbelievable slow.</div><div><br></div><div>Apparently SCTP SACKs are only send after the application has actually received (on the recv syscall) the payload.</div><div>So,what I'm seeing is that if your application takes to long to read the payload, the SACK will not be send and the sender will start retransmitting the SCTP packet.<br></div><div><br></div><div>I have written a simple socket benchmark client [1].</div><div><br></div><div>When testing with TCP, UDP and SCTP on OTP 21.1, sending 1000 packets with a length of 100 bytes, I get this results:</div><div><div><br></div><div>* server is socat to /dev/null, e.g. "socat -u SCTP-LISTEN:6000,reuseaddr,keepalive,rcvbuf=131071,reuseaddr /dev/null")</div><div class="gmail_quote"><div>* times are in microseconds</div><br class="inbox-inbox-Apple-interchange-newline"></div></div><div><div>$ ./bench.escript sctp 127.0.0.1 6000 100 1000<br></div><div>sctp,100,1000,817250<br></div><div>tcp,100,1000,8301<br></div><div>udp,100,1000,17477<br></div><div><br></div><div>As you can see TCP is fastest with ~8ms due to the Nagle algorithm combining the packets before sending, UDP is still ok with ~17.5ms. SCTP takes a astonishing 817ms. That is 46 times slower than UDP.</div><div><br></div><div>Regards</div><div>Andreas</div><div><br></div><div>[1]: <a href="https://gist.github.com/RoadRunnr/53c19861aa4e4fa5bd45c072727ab971">https://gist.github.com/RoadRunnr/53c19861aa4e4fa5bd45c072727ab971</a></div><div><br></div><div>Oliver Korpilla <<a href="mailto:Oliver.Korpilla@gmx.de">Oliver.Korpilla@gmx.de</a>> schrieb am Di., 13. Nov. 2018 um 18:01 Uhr:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello, Jesper.<br>
<br>
The problem I see that the C++ side just fails to send more messages back but I'm stumped why.<br>
<br>
It _looks_ like it fails to respond to my protocol requests for some reason.<br>
<br>
But does it really? Or is something blocking/buffering/delaying/missing in the stack? And which side causes it?<br>
<br>
I'm very very stumped. Because I've seen the tcpdump in Wireshark and C++ stops sending. It just stops. (If I had more trust in my SCTP knowledge I would _assume_ there's some sort of deadlock on the C++ side.)<br>
<br>
Thank you very much,<br>
Oliver <br>
<br>
Gesendet: Dienstag, 13. November 2018 um 15:51 Uhr<br>
Von: "Jesper Louis Andersen" <<a href="mailto:jesper.louis.andersen@gmail.com" target="_blank">jesper.louis.andersen@gmail.com</a>><br>
An: "Oliver Korpilla" <<a href="mailto:Oliver.Korpilla@gmx.de" target="_blank">Oliver.Korpilla@gmx.de</a>><br>
Cc: "Erlang (E-mail)" <<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><br>
Betreff: Re: [erlang-questions] gen_sctp: What delays SACK?<br>
<br>
Hi,<br>
<br>
Use tcpdump(1) on the flow and look for who is adding the latency. Usual rule of protocol debugging is to start at the lowest level and verify each level as you go up. Because then you have an audit trail of the events that happened which can inform you at a higher level.<br>
<br>
<br>
On Tue, Nov 13, 2018 at 10:15 AM Oliver Korpilla <<a href="mailto:Oliver.Korpilla@gmx.de" target="_blank">Oliver.Korpilla@gmx.de</a>[mailto:<a href="mailto:Oliver.Korpilla@gmx.de" target="_blank">Oliver.Korpilla@gmx.de</a>]> wrote:Hello.<br>
<br>
We're using an elixir application as a sort of protocol tester. It communicated with the system-under-test over SCTP as a transport.<br>
<br>
We're observing delay and unsent messages and due to the nature of the SCTP protocol we're not sure which side causes the issue.<br>
<br>
The BEAM side has the NO_DELAY option set and pumps a burst of messages but then waits for responses (so it will not burst indefinitely, it burst once and then respond).<br>
<br>
The C++ application has the DELAYED_SACK option set - we tried with both sack_freq 1 (which supposedly disables the algorithm) and higher (the default in our system).<br>
<br>
(We also increased the receive window on both sides to ensure that senders would not block.)<br>
<br>
But we're stumped. The C++ side is not responding at some point. When we did an actual target test once and we saw SCTP messages sent from system-under-test just stop when analyzing the tcpdump of the interfaces - C++ application has not emitted something on the wire and respectively nothing is received.<br>
<br>
<br>
Our latest area of inquiry is to find out if maybe the elixir part is simply not getting scheduled - but can this impact for example SACK latency? Who acknowledges a message - the SCTP stack by itself or the application? And will the protocol block the sender until SACK?<br>
<br>
I'm sorry for asking such vague questions but SCTP know-how is spread thin in our outfit and we're not the experts...<br>
<br>
Thank you,<br>
Oliver<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>[mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>]<br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a> <br>
--<br>
J.<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span style="font-family:monospace,monospace">-- <br>Dipl.-Inform. Andreas Schultz<br><br>----------------------- enabling your networks ----------------------<br>Travelping GmbH Phone: +49-391-81 90 99 0<br>Roentgenstr. 13 Fax: +49-391-81 90 99 299<br>39108 Magdeburg Email: <a href="mailto:info@travelping.com">info@travelping.com</a><br>GERMANY Web: <a href="http://www.travelping.com">http://www.travelping.com</a><br><br></span><div><span style="font-family:monospace,monospace">Company Registration: Amtsgericht Stendal Reg No.: HRB 10578</span></div><span style="font-family:monospace,monospace">Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780<br>---------------------------------------------------------------------</span></div></div>