<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 6 apr 2012, at 15:23, Per Hedeland wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Tony Rogvall <<a href="mailto:tony@rogvall.se">tony@rogvall.se</a>> wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">First of all the right way: send, shutdown wait for close is not working here as would have been <br></blockquote><blockquote type="cite">my first answer :-) <br></blockquote><br>Hm, why do you (too) bring shutdown into this discussion? Surely you are<br>not suggesting that it should be a requirement to call<br>gen_tcp:shutdown/2 before gen_tcp:close/1 in order to get buffered data<br>delivered? The gen_tcp interface is a very nice simplification of the<br>low-level socket API - why should it be *more* complex than the<br>low-level API in this particular case?<br><br></div></blockquote><div>I was only trying an alternative to close that would not stop the reading side just to check</div><div>if shutdown had the same problem. And it had. Simple as that.</div><div><br></div><blockquote type="cite"><div>I will maintain that the only reason to use shutdown is when you want to<br>do a "half close", or even only when you want to do a "half close in the<br>send direction". What's the point otherwise?<br><br></div></blockquote><div>Of course, I do not see your point ?</div>I will maintain that if you want to be sure that the other side actually got all your data</div><div>then use shutdown followed by "wait for close". </div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite">Then I moved the subscription code to just wait for the empty queue event, and NOT do the timeout. <br></blockquote><blockquote type="cite">This worked well.<br></blockquote><br>But will potentially block forever.<br><br></div></blockquote>I know that! I was still looking for the "bug".</div><div><br><blockquote type="cite"><div><blockquote type="cite">My interpretation is that the reader is so slow, so that the write side <br></blockquote><blockquote type="cite">"write ready" is not signaled and no more data goes down to kernel space (probably some threshold.)<br></blockquote><blockquote type="cite">This will keep the pending data at the same level, and hence timeout.<br></blockquote><br>Agreed.<br><br><blockquote type="cite">So how would a workaround look like ? <br></blockquote><br>IMO the *fix* is to not have a user-level queue at all, and thus always<br>have gen_tcp:send/2 block until everything has been written to the<br>socket. I don't expect this fix to happen though, but it should at least<br>be possible to disable the user-level queue, by passing options<br>{high_watermark, 1}, {low_watermark, 0}. However this doesn't work the<br>way the code is currently written - send() will never block if the queue<br>is empty before the call. The patch below fixes this, and with it I can<br>run Matt's test successfully (after fixing some "bad" assumptions) if I<br>use the watermark-setting.<br><br></div></blockquote>Cool. Lets see if this "remove inet_drv queue" is reasonable, did you</div><div>check the effects on erlang distribution ?</div><div><br></div><div><blockquote type="cite"><div><blockquote type="cite">One way is to make sure that the kernel buffer is smaller (now I can send 160K). Use what ever size that<br></blockquote><blockquote type="cite">make sense with the knowledge of the inet built in time out timer (5 secs)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">An other way to fix this could be to try to add kernel buffer space to send_pend but that is <br></blockquote><blockquote type="cite">pretty os dependent :-)<br></blockquote><br>Both of these suffer from the effect of "silly window avoidance" - i.e.<br>even if they improve the coupling "more data is sent" => "send queue<br>shrinks", they do not help with the coupling "more data is read" =><br>"more data is sent". And they don't at all address the "keep trying as<br>long as the receiver is alive" goal.<br><br></div></blockquote>Is there any way of mimic the kernel in this sense ?</div><div><br></div><div>/Tony</div><div><br><blockquote type="cite"><div>--Per<br><br><br>--- otp_src_R15B01/erts/emulator/drivers/common/inet_drv.c.ORIG<span class="Apple-tab-span" style="white-space:pre"> </span>2012-04-01 20:15:00.000000000 +0200<br>+++ otp_src_R15B01/erts/emulator/drivers/common/inet_drv.c<span class="Apple-tab-span" style="white-space:pre"> </span>2012-04-06 13:18:59.000000000 +0200<br>@@ -9396,6 +9396,19 @@<br> return -1;<br> }<br><br>+static void tcp_set_busy(tcp_descriptor* desc)<br>+{<br>+ DEBUGF(("tcp_sendv(%ld): s=%d, sender forced busy\r\n",<br>+ (long)desc->inet.port, desc->inet.s));<br>+ desc->inet.state |= INET_F_BUSY; /* mark for low-watermark */<br>+ desc->inet.busy_caller = desc->inet.caller;<br>+ set_busy_port(desc->inet.port, 1);<br>+ if (desc->send_timeout != INET_INFINITY) {<br>+ desc->busy_on_send = 1;<br>+ driver_set_timer(desc->inet.port, desc->send_timeout);<br>+ }<br>+}<br>+<br> /*<br> ** Send non-blocking vector data<br> */<br>@@ -9439,15 +9452,7 @@<br> if ((sz = driver_sizeq(ix)) > 0) {<br> <span class="Apple-tab-span" style="white-space:pre"> </span>driver_enqv(ix, ev, 0);<br> <span class="Apple-tab-span" style="white-space:pre"> </span>if (sz+ev->size >= desc->high) {<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> DEBUGF(("tcp_sendv(%ld): s=%d, sender forced busy\r\n",<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span> (long)desc->inet.port, desc->inet.s));<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> desc->inet.state |= INET_F_BUSY; /* mark for low-watermark */<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> desc->inet.busy_caller = desc->inet.caller;<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> set_busy_port(desc->inet.port, 1);<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> if (desc->send_timeout != INET_INFINITY) {<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>desc->busy_on_send = 1;<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>driver_set_timer(desc->inet.port, desc->send_timeout);<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> }<br>+ tcp_set_busy(desc);<br> <span class="Apple-tab-span" style="white-space:pre"> </span> return 1;<br> <span class="Apple-tab-span" style="white-space:pre"> </span>}<br> }<br>@@ -9490,8 +9495,13 @@<br> <span class="Apple-tab-span" style="white-space:pre"> </span>DEBUGF(("tcp_sendv(%ld): s=%d, Send failed, queuing\r\n", <br> <span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>(long)desc->inet.port, desc->inet.s));<br> <span class="Apple-tab-span" style="white-space:pre"> </span>driver_enqv(ix, ev, n); <br>-<span class="Apple-tab-span" style="white-space:pre"> </span>if (!INETP(desc)->is_ignored)<br>+<span class="Apple-tab-span" style="white-space:pre"> </span>if (!INETP(desc)->is_ignored) {<br> <span class="Apple-tab-span" style="white-space:pre"> </span> sock_select(INETP(desc),(FD_WRITE|FD_CLOSE), 1);<br>+ if (driver_sizeq(ix) >= desc->high) {<br>+ tcp_set_busy(desc);<br>+ return 1;<br>+ }<br>+ }<br> }<br> return 0;<br> }<br>@@ -9536,15 +9546,7 @@<br> <span class="Apple-tab-span" style="white-space:pre"> </span> driver_enq(ix, buf, h_len);<br> <span class="Apple-tab-span" style="white-space:pre"> </span>driver_enq(ix, ptr, len);<br> <span class="Apple-tab-span" style="white-space:pre"> </span>if (sz+h_len+len >= desc->high) {<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> DEBUGF(("tcp_send(%ld): s=%d, sender forced busy\r\n",<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span> (long)desc->inet.port, desc->inet.s));<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> desc->inet.state |= INET_F_BUSY; /* mark for low-watermark */<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> desc->inet.busy_caller = desc->inet.caller;<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> set_busy_port(desc->inet.port, 1);<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> if (desc->send_timeout != INET_INFINITY) {<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>desc->busy_on_send = 1;<br>-<span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>driver_set_timer(desc->inet.port, desc->send_timeout);<br>-<span class="Apple-tab-span" style="white-space:pre"> </span> }<br>+ tcp_set_busy(desc);<br> <span class="Apple-tab-span" style="white-space:pre"> </span> return 1;<br> <span class="Apple-tab-span" style="white-space:pre"> </span>}<br> }<br>@@ -9590,8 +9592,13 @@<br> <span class="Apple-tab-span" style="white-space:pre"> </span> n -= h_len;<br> <span class="Apple-tab-span" style="white-space:pre"> </span> driver_enq(ix, ptr+n, len-n);<br> <span class="Apple-tab-span" style="white-space:pre"> </span>}<br>-<span class="Apple-tab-span" style="white-space:pre"> </span>if (!INETP(desc)->is_ignored)<br>+<span class="Apple-tab-span" style="white-space:pre"> </span>if (!INETP(desc)->is_ignored) {<br> <span class="Apple-tab-span" style="white-space:pre"> </span> sock_select(INETP(desc),(FD_WRITE|FD_CLOSE), 1);<br>+ if (driver_sizeq(ix) >= desc->high) {<br>+ tcp_set_busy(desc);<br>+ return 1;<br>+ }<br>+ }<br> }<br> return 0;<br> }<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; ">"Installing applications can lead to corruption over time. </span><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; ">Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"</span></div><div><span class="Apple-style-span" style="color: rgb(51, 51, 51); font-family: Geneva, Arial, Helvetica, sans-serif; font-size: 12px; "><br></span></div></span><br class="Apple-interchange-newline">
</div>
<br></body></html>