<div dir="ltr">Hi Edwin,<br><br>On my main test machine I already had set the MaxUserPort to 60000 from a former unrelated test. It does not change the problem.<br>The crashes occure way before all ephemeral ports are in TIME_WAIT.<br>
<br>So, I am still able to crash:<br>a) the client application when run without server<br>b) the client when run together with the server<br>c) the server when run together with the client<br>d) even both at the same time<br>
started under erl.exe it takes significant more processes than started under werl.exe.<br><br>I will send a mail to erlang-bugs.<br><br>Thank you for taking the time and helping me!<br><br>Regards,<br>Michael<br><br><div class="gmail_quote">
On Sat, Sep 13, 2008 at 5:49 AM, Edwin Fine <span dir="ltr"><<a href="mailto:erlang-questions_efine@usa.net">erlang-questions_efine@usa.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">Micheal,<br><br>I was able to duplicate your problem on WIndwos XP <b>SP3</b> ;) on an Intel E6600 dual-core system in 32-bit mode with 4 GB of RAM, Erlang R12B-3, but it took more processes than 1000 (I used 10000), at which point the emulator crashed with the same error message you reported. It crashed in the second test, but I didn't try to get the first test to crash after that.<br>
<br>At the very least, this should be reported as a bug to erlang-bugs because the crash dump report is unhelpful.<br><br>I wonder if you are running into an ephemeral port starvation (TIME_WAIT) problem? if you run netstat in a command window you will most likely see many ports in a TIME_WAIT state. One way to find out is to increase the number of ephemeral ports and see if that delays or eliminates the problem unless you run with more processes, or run more often.<br>
<br>A good explanation about the TIME_WAIT state can be found at<br><br><a href="http://www.developerweb.net/forum/showthread.php?t=2941" target="_blank">http://www.developerweb.net/forum/showthread.php?t=2941</a><br><br>
On Linux:<br># Set TIME_WAIT timeout to 30 seconds instead of 120<br>
/sbin/sysctl -w net.ipv4.tcp_fin_timeout=30<br><br>On Windows:<br>In the registry, under HKEY_LOCAL_MACHINE\SYSTEM\<div>CurrentControlSet\Services\Tcpip\Parameters<ul><li>Add DWORD MaxUserPort = 65534 (decimal).</li>
<li>Add TcpTimedWaitDelay = 60 (decimal)<br>
</li></ul>Explanation:<br><br>The default maximum number of <span>ephemeral</span> TCP ports is 5000. Increase the maximum number of <span>ephemeral</span>
ports (MaxUserPort) to 65534 (decimal). This parameter controls the
maximum port number that is used when a program requests any available
user port from the system. Typically , <span>ephemeral</span>
(short-lived) ports are allocated between the values of 1024 and 5000
inclusive. This parameter determines the time that a connection stays
in the TIME_WAIT state when it is closing. As long as a connection is
in the TIME_WAIT state, the socket pair cannot be re-used. This is also
known as the "2MSL" state. According to RFC793, the value should be two
times the maximum segment lifetime on the network. See RFC793 for more
information. However, in fast LAN environments (same-segment 1GB/sec,
for example), this can be lowered to 60 or less.<br><br>
TcpTimedWaitDelay - Set to 60<br>
<div>
Key: Tcpip\Parameters<br>
Value Type: REG_DWORD - Time in seconds<br>
Valid Range: 30-300 (decimal)<br><br></div></div>Hope this helps.<div><div></div><div class="Wj3C7c"><br><br><div class="gmail_quote">On Fri, Sep 12, 2008 at 6:20 PM, Edwin Fine <span dir="ltr"><<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div dir="ltr">My mistake, thanks.<div><div></div><div><br><br><div class="gmail_quote">
On Fri, Sep 12, 2008 at 5:42 PM, Michael Regen <span dir="ltr"><<a href="mailto:michael.regen@gmail.com" target="_blank">michael.regen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div dir="ltr">I would be very interested to hear about some tests from others! And Edwin, I guess you mean SP3. Windows XP SP3 is the most recent service pack. SP4 is if at all a giant trojan, isn't it? ;)<br>
<br>Regards,<br><font color="#888888">
Michael</font><div><div></div><div><br><br><div class="gmail_quote">On Fri, Sep 12, 2008 at 9:01 PM, Edwin Fine <span dir="ltr"><<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">I hear you :).<br><br>When I've got some time I will try out my R12B-3 installation on SP4 with your program, but can't right now. I am interested to see what happens.<br><br>Rgds,<br>Ed<div><div>
</div>
<div><br><br><div class="gmail_quote">
On Fri, Sep 12, 2008 at 2:47 PM, Michael Regen <span dir="ltr"><<a href="mailto:michael.regen@gmail.com" target="_blank">michael.regen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">Just tested it on another machine with SP3 installed. No difference. Same problem.<br><br>Yes, Windows is flaky and I personally would like to be able to say I would rather go to hell than installing a server on Windows which is expected to run robust.<br>
But well, either Erlang is robust on Windows as well or no Erlang on Windows. :(<br><br>Regards,<br>Michael<br><font color="#888888"><br>-- <br>Quote from a >3000 employees IT centric company's CIO I had the pleasure to witness four weeks ago: 'For the messaging back end? No, we can't use Java. Java is too slow.'</font><div>
<div></div><div><br>
<br><br><br><div class="gmail_quote">On Fri, Sep 12, 2008 at 7:54 PM, Edwin Fine <span dir="ltr"><<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">Michael,<br><br>I've always felt that the Windows version of Erlang is a bit flaky. Then again, I think Windows itself is more than a bit flaky, so maybe it's not Erlang's fault ;)<br>I wonder if running on SP4 would improve things?<div>
<div></div><div><br>
<br><div class="gmail_quote">On Fri, Sep 12, 2008 at 1:39 PM, Michael Regen <span dir="ltr"><<a href="mailto:michael.regen@gmail.com" target="_blank">michael.regen@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">Hi Edwin,<br><br>It is possible that both issues have a similar source but I do not see many reasons why there must be a common source.<br>I was running my tests on a 32bit single core Windows XP SP2 system just by running<br>
werl.exe -boot start_sasl<br>or<br> werl.exe<br><br>and did nothing fancy. My R12B-3 version is self compiled, R12B-4 is out of the <a href="http://erlang.org" target="_blank">erlang.org</a> box.<br>Client and server tests where done by starting two different instances of werl.<br>
Furthermore my tcp_test:test does not care whether results from gen_tcp:connect are correct or not. It just assumes {ok, Socket} and crashes the process if otherwise. Of course it was a surprise that under some circumstances the whole emulator crashes.<br>
<br>By the way, the crash dump slogan is unspectecular: 'Slogan: Inconsistent, why isnt io reported?'<br><br>UPDATE: I got some more observations which puzzle me even more:<br><br>Just did some more of the same tests but this time by starting:<br>
erl.exe<br> application:start(tcp_server).<br>and<br> erl.exe<br> tcp_test:test(1000).<br><br>There seems to be a difference between erl.exe and werl.exe.<br><br>This time results are pretty different:<br>Now it is much harder to crash the emulator. It takes significant more processes / tries until something bad happens:<br>
<br>client only (tcp_test:test(5000)) crashes eventually in the same way but Window's cmd.exe now follows with a:<br> The exception unknown software exception (0x40000015) occured in the application at location 0x008fff86<br>
<br>after the 'Crash dump was written to: erl_crash.dump / Inconsistent, why isnt io reported?' message and the crash dump file.<br><br>The exception seems to always occure at the same location.<br><br>A lot more error messages are printed now (as expected) until the crash.<br>
Besides the {{badmatch,{error,econnrefused}},[{tcp_test,test_con,0}]} I can now also watch lots of<br> {{badmatch,{error,eaddrinuse}},[{tcp_test,test_con,0}]}<br>and<br> {{badmatch,{error,system_limit}},[{tcp_test,test_con,0}]}<br>
errors.<br><br>The good message: During tests together with the server backend I was not able to crash the server. But I am not convinced that erl.exe solves everthing server side.<br><br>Regards,<br><font color="#888888">Michael</font><div>
<div></div><div><br><br><div class="gmail_quote">
On Fri, Sep 12, 2008 at 6:32 PM, Edwin Fine <span dir="ltr"><<a href="mailto:emofine@gmail.com" target="_blank">emofine@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr">Please be aware that I reported a bug a while ago on erlang-bugs, where attempting to connect to a socket that is not being listened on will sometimes return an actual success return, but subsequent operations will fail. Here is an excerpt from that bug report.<br>
<br><pre>When calling gen_tcp:connect/3 or /4 on a host/port that does not have a<br>running program listening on it, at random intervals gen_tcp:connect returns<br>an {ok, Sock} instead of the expected {error, econnrefused}. If<br>
gen_tcp:recv(Sock, 0) is called immediately using the socket just returned,<br>it returns an {error, econnrefused}. Connection options used were [binary,<br>{packet, raw}, {active, false}]. It should be noted that the gen_tcp:connect<br>
succeeds when there is a program listening on that sane host/port, so it's<br>unlikely to be a firewall issue.<br></pre>See <a href="http://www.erlang.org/pipermail/erlang-bugs/2008-August/000931.html" target="_blank">http://www.erlang.org/pipermail/erlang-bugs/2008-August/000931.html</a><br>
<br>This bug is still present in R12B-4. Could this be affecting you?<br><br>Regards,<br>Edwin Fine<br><br><div class="gmail_quote">2008/9/12 Michael Regen <span dir="ltr"><<a href="mailto:michael.regen@gmail.com" target="_blank">michael.regen@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div><div dir="ltr">Hi,<br><br>I got a series of troubles with gen_tcp all eventually resulting in crashes. I tested this under Windows XP and with R12B-3 as well as R12B-4. Under Linux it seems to work but I am not perfectly sure since the crash happens sporadically and seems to be timing related.<br>
<br>The two problems below lead me to a couple of questions:<br>a) What is the real cause? Is it the socket error enfile? Do both problems have the same root cause?<br>b) Is there a bug in Erlang? I guess this should not lead to a crash.<br>
c) How do you avoid this problem on systems you do not control yourself?<br><br><br>Problem #1:<br>###########<br><br>Just compile the following code and run it with sasl enabled and the following command:<br> tcp_test:test(1000).<br>
and - yes - without anything listening on port 2222. And sometimes you have to try two times!<br><br>-------------------------- start: tcp_test.erl --------------------------<br>-module(tcp_test).<br><br>-export([test/1, test_con/0]).<br>
<br>-define(DEF_PORT, 2222).<br>-define(DEF_IP, {127,0,0,1}).<br><br>test(0) -> ok;<br>test(HowManyProcs) -><br> spawn(?MODULE, test_con, []),<br> test(HowManyProcs-1).<br><br>test_con() -><br> {ok,S} = gen_tcp:connect(?DEF_IP, ?DEF_PORT,[]),<br>
gen_tcp:send(S,<<0,5,65,66,67,68,69>>),<br> receive<br> {tcp_closed, _Socket} -> ok;<br> _Msg -> gen_tcp:close(S)<br> after 500 -><br> gen_tcp:close(S)<br> end.<br>-------------------------- end: tcp_test.erl --------------------------<br>
<br>It just spawns a bunch of processes all trying to connect to a currently closed port and sending some garbage there. This is what happens:<br><br>-------------------------- start: log tcp_test.erl --------------------------<br>
=ERROR REPORT==== 12-Sep-2008::15:28:47 ===<br>Error in process <0.41.0> with exit value: {{badmatch,{error,econnrefused}},[{tcp_test,test_con,0}]}<br><br>[... a couple of them but usually between 1 and 20.]<br><br>
=ERROR REPORT==== 12-Sep-2008::15:28:47 ===<br>Error in process <0.103.0> with exit value: {{badmatch,{error,econnrefused}},[{tcp_test,test_con,0}]}<br><br><br>Crash dump was written to: erl_crash.dump<br>Inconsistent, why isnt io reported?<br>
<br>Abnormal termination<br>-------------------------- end: log tcp_test.erl --------------------------<br><br>It might have something to do with the socket error enfile 'file table overflow' but I guess it should not simply crash the emulator!?<br>
Searching google for 'Inconsistent, why isnt io reported?' just gives one hit to Erlang's source code.<br>I can provide the crash dump if needed. Just did not want to spam the whole list with big attachments.<br>
Spawning only 500 processes (tcp_test:test(500).) usually leads to a crash, spawning only 200 seems to work.<br><br><br>Problem #2:<br>###########<br><br>Now let's try the same with a server answering to port 2222: Just take the code from the trapexit tutorial 'Building a Non-blocking TCP server using OTP principles' <a href="http://trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles" target="_blank">http://trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles</a><br>
Start it first and then our test module in a different erlang node as described above. Now, usually the client survives (have seen crashes as well!) and the server crashes in a similar way. Sometimes it survives and in very rare cases you will see the following logs in the erlang server instance:<br>
<br>-------------------------- start: log server --------------------------<br>=ERROR REPORT==== 12-Sep-2008::12:58:56 ===<br>File operation error: system_limit. Function: get_cwd. Process: code_server.<br><br>=ERROR REPORT==== 12-Sep-2008::12:58:56 ===<br>
Error in async accept: {async_accept,"file table overflow"}.<br><br>=ERROR REPORT==== 12-Sep-2008::12:58:56 ===<br>** Generic server tcp_listener terminating <br>** Last message in was {inet_async,#Port<0.109>,1019,{ok,#Port<0.2141>}}<br>
** When Server state == {state,#Port<0.109>,1019,tcp_echo_fsm}<br>** Reason for termination == <br>** {async_accept,"file table overflow"}<br><br>[...]<br>-------------------------- end: log server --------------------------<br>
<br>Can anyone help? Thank you!<br><br>Regards,<br>Michael<br></div>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>