<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 6 apr 2012, at 12:39, Per Hedeland wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Matthias Lang <<a href="mailto:matthias@corelatus.se">matthias@corelatus.se</a>> wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">On Tuesday, April 03, Andreas Schultz wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">In my case the receiver is to slow to process all the data, sender<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">does 10k packets of 1k size, the receiver only gets the first 2000<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">packets. It might well be that I hit the close timeout and inet<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">discards the rest of the send queue.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Ok, that's something to go on.<br></blockquote><br>And I note that we have moved from the initial suspicion of "buffered<br>data is not flushed out, but simply discarded" to an observation of the<br>flushing doing a less than stellar job in specific circumstances. But<br>I'll admit to being unaware that it did such a bad job in circumstances<br>that aren't extremely unusual. Those circumstances being roughly "our<br>sending is so far ahead of the receiver's consumption of data that a)<br>the TCP window is closed, b) the kernel-level socket send buffer is<br>full, and c) inet_drv has written data into its user-level queue".<br><br></div></blockquote>I had to do some digging myself to realize the same fact :-)</div><div><br></div><div><br><blockquote type="cite"><div><blockquote type="cite">I had a bit of a dig in prim_inet.erl. It sounds like you've looked<br></blockquote><blockquote type="cite">there too. That code looks like it's intended to loop 'forever' trying<br></blockquote><blockquote type="cite">to send the queued data, as long as some progress is made every so often<br></blockquote><blockquote type="cite">(always sends at least something in every 5s timeout period). But running<br></blockquote><blockquote type="cite">my program suggests that isn't happening as intended.<br></blockquote><br>Yes, that code looks sort-of reasonable (modulo the bug you possibly<br>found), but it isn't really. I think we want to define "progress" as<br>"the receiver read *something* off his end of the connection". But the<br>check for progress is the size of the user-level queue, i.e. c) above,<br>and that is pretty far removed from this definition of progress: Reading<br>of data is not immediately reflected in opening of the TCP window (see<br>"silly window syndrome" - this is true for any non-broken TCP<br>implementation), and sending of data from the socket send buffer, once<br>the window *has* opened, may not immediately trigger poll(POLLOUT) (this<br>is true at least for the Linux version I experimented with, but is<br>probably common) - and this is what is required for the user-level queue<br>to shrink.<br><br></div></blockquote>That is pretty much what I found as well.</div><div><br></div><div><br><blockquote type="cite"><div>Finally, requiring progress at all is a rather arbitrary decision. We<br>might want to keep trying as long as we have evidence that the receiver<br>is even alive and possibly willing to read more at some point in the<br>future. This is what TCP/kernel does when you close(2) the OS-level TCP<br>socket with unsent data - even if the TCP window remains closed, it will<br>retain the buffer and keep sending window probes, as long as those<br>probes elicit ACKs from the other end. In fact this behavior is pretty<br>much spelled out in the venerable RFC 793, so it could be argued that<br>gen_tcp violates the TCP spec when it "gives up" on the close.<br><br></div></blockquote></div><div>Maybe reading some stats about the socket to mimic the TCP behavior is</div><div>needed then? </div><div><br></div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite"><blockquote type="cite">The fix should be simple, limit the send queue size.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">To what?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Zero seems to be the only value that will work even for arbitrarily slow<br></blockquote><blockquote type="cite">clients. And that defeats the point of having a send queue.<br></blockquote><br>Exactly - which raises the question, what *is* the point of having a<br>send queue? I.e. the user-level queue that inet_drv maintains, and which<br>is causing these problems. I don't know, but since Tony has joined the<br>discussion, maybe he can answer.:-)<br><br></div></blockquote>I remember us hacking a proxy more that 10 years ago, then we had to tweak sndbuf size </div><div>inorder to keep Windows not killing buffers with RST.Iif I remember correctly we also had</div><div>to keep a low memory footprint per connection in order to have plenty of them.</div><div>(too many dialup clients at that time)</div><div><br></div><div>Remember that inet_drv is not only used by gen_tcp but is also used by distribution.</div><div>The internal port queues are a great place to push stuff when you should be blocking,</div><div>but are not really ready to do that yet :-)</div><div><br></div><div>To push wouldblock back to Erlang could be a way of handling the problem</div><div>and let Erlang get the POLLOUT signal etc. But WE did not design it that way :-)</div><div><br></div><div>Plenty of history around the inet_drv, not saying it is doing the correct thing or even close</div><div>but it is an explanation.</div><div><br></div><div><br><blockquote type="cite"><div><blockquote type="cite">It's late, I might have outsmarted myself, but my current feeling is<br></blockquote><blockquote type="cite">that erlang is quietly tossing data and it shouldn't be.<br></blockquote><br>I agree.<br><br><blockquote type="cite">Waiting for as long as it takes in close() seems like the right thing,<br></blockquote><blockquote type="cite">though Per might disagree.<br></blockquote><br>Well, as I wrote earlier, I don't expect close() to block until all data<br>is sent - in fact I don't expect it to block at all. We already have a<br>potentially-blocking send() call, with an optional timeout even (unlike<br>close()), why shouldn't that be enough? Your suggestion should get the<br>job done, but it would block until the user-level queue has drained,<br>which may in principle be forever.<br><br></div></blockquote>While data is queued in user land and some stop the node (init:stop or ^C ...)</div><div>This data will be discarded, while if data made it into kernel it will not, </div><div>I do not see how this can be fixed (except using queues in user land)</div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite">Waiting for N seconds in close() and then<br></blockquote><blockquote type="cite">returning an error if the queue didn't empty would also be better than<br></blockquote><blockquote type="cite">just quietly tossing it.<br></blockquote><br>Maybe - but when you close(2) the OS-level socket (without SO_LINGER),<br>you are anyway saying that you don't want to be informed about the final<br>outcome, and if the receiver dies before reading all the data, you won't<br>be told. I don't think it makes a lot of sense to inform specifically<br>about the failure to drain the user-level queue - it might even give the<br>false impression that you will always be told about failures.<br><br><blockquote type="cite">(And: yes, I know, application-level ACKs would avoid this<br></blockquote><blockquote type="cite">problem. But I'm not quite ready to say that this problem can't be<br></blockquote><blockquote type="cite">fixed.)<br></blockquote><br>I'm tempted to refer to the FAQ entry about Erlang message passing where<br>you quote other ramblings of mine - how much reliability do you want?<br>If someone's life depends on the receiver consuming all the data and<br>acting on it, you'd better have application-level ACKs despite the fact<br>that TCP (like Erlang message passing) is "reliable". If it's just a<br>question about whether a jpeg gets displayed in a web browser, probably<br>not.<br><br>But it's reasonable to expect that if the networks, hosts, and<br>applications keep running, and the user doesn't close the tab in his<br>browser, that jpeg *should* eventually be displayed in full even if the<br>user is on a slow dialup and the sender has long since completed his<br>close() call and gone on to other business (or even called it a day).<br>gen_tcp:close/1 doesn't meet this expectation.<br><br></div></blockquote>Agreed.</div><div><br></div><div>/Tony</div><div><br><blockquote type="cite"><div>--Per<br>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>http://erlang.org/mailman/listinfo/erlang-questions<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; ">"Installing applications can lead to corruption over time. </span><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; ">Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"</span></div><div><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; "><br></span></div></span><br class="Apple-interchange-newline">
</div>
<br></body></html>