<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Yeah you got that right ! leaking at a huge rate at some point !</div><div><br></div><div>- The number of Fd - I don't get close to the max</div><div><div># cat /proc/sys/fs/file-nr</div><div>3264 0 6455368</div></div><div><br></div><div>- On the production server there is only the erlang node, no other service…</div><div>The beam.smp was through the roof at 300% CPU and 97% RAM</div><div>The weird thing is that it got there in a second, I was looking at it when it happens.</div><div><br></div><div>- It has happened with 2000 connections, 4000 connections, and 10000 connections… 5 min after start, 5hours after start.</div><div><br></div><div>I really can't find a pattern here…and I'm becoming a little desperate.</div><div><br></div><div>Thank you for your help again.</div><div><br></div><div>Morgan.</div><br><div><div>Le 23 mai 2013 à 23:20, Dmitry Kolesnikov <<a href="mailto:dmkolesnikov@gmail.com">dmkolesnikov@gmail.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>You system definitely leaking some resources :-/</div><div> - Check number of used FD(s) may be you exceeded limit there </div><div> - What was overall system memory / cpu utilisation before crash?</div><div> - Check how many connections you got before crash, may be you can reproduce it at dev</div><div><br></div><div>- Dmitry</div><br><div><div>On May 24, 2013, at 12:13 AM, Morgan Segalis <<a href="mailto:msegalis@gmail.com">msegalis@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Ok, it finally got into the infinite loop…</div><div><br></div><div>And of course, the node on which I was running etop could not give me anymore since it got disconnected from the production node.</div><div><br></div><div>So back to square one… no way to investigate correctly so far :-/</div><div><br></div><div>Morgan.</div><br><div><div>Le 23 mai 2013 à 16:34, Morgan Segalis <<a href="mailto:msegalis@gmail.com">msegalis@gmail.com</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Yeah that what I'm doing right now, but of course, when I'm monitoring it, it won't crash, only when I sleep !!<div><br></div><div>I get back to the Erlang list as soon as I have more informations about this.</div><div><br></div><div>Thank you all !</div><div><br></div><div>Morgan.</div><div><br><div><div>Le 23 mai 2013 à 16:30, Vance Shipley <<a href="mailto:vances@motivity.ca">vances@motivity.ca</a>> a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><p dir="ltr">Keep etop running and capture the output to a file (e.g. etop ... | tee stop.log). After it gets into trouble look back and see what was happening beforehand.<br>
</p>
<div class="gmail_quote">On May 23, 2013 6:16 PM, "Morgan Segalis" <<a href="mailto:msegalis@gmail.com">msegalis@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">So I should go back to R15B ?<div><br></div><div>erlang:memory() gives me </div><div><br></div><div><div>[{total,1525779584},</div><div> {processes,1272881427},</div><div> {processes_used,1272789743},</div>
<div> {system,252898157},</div><div> {atom,372217},</div><div> {atom_used,346096},</div><div> {binary,148093608},</div><div> {code,8274446},</div><div> {ets,1546832}]</div></div><div><br></div><div><br></div><div>But keep in mind that right now, there is no infinite loop, or memory issue at this exact time…</div>
<div>It will be more interesting to have that when the VM is asking for 14GB of memory, but when it does, the console is unresponsive, so I can't get anything then.</div><div><br><div><div>Le 23 mai 2013 à 14:39, Dmitry Kolesnikov <<a href="mailto:dmkolesnikov@gmail.com" target="_blank">dmkolesnikov@gmail.com</a>> a écrit :</div>
<br><blockquote type="cite"><div style="word-wrap:break-word"><div>Right, you do not have many processes. Same time you goes out of memory…</div><div><br></div><div>Unfortunately, I had no time play around with R16B at production… </div>
<div>Could it be some issue with SSL, I re-call there was some complains in the list? </div><div><br></div><div>I would use entop to spot the process that has either too much reductions, queue len or heap.</div><div>Once you know they pid you can dig more info about them using erlang:process_info(…) and/or sys:get:status(…)</div>
<div><br></div><div>BTW, What erlang:memory() says on you production node?</div><div><br></div><div>- Dmitry</div><br><div><div>On May 23, 2013, at 3:25 PM, Morgan Segalis <<a href="mailto:msegalis@gmail.com" target="_blank">msegalis@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div style="word-wrap:break-word"><div>No, I was talking about the function I made to investigate which processes I have created, which gives me this output : </div><div><br></div><div>Dict: {dict,16,16,16,8,80,48,<br>
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},<br> {{[[{{connector_serv,init,1},[connector_suprc42,connector,<0.42.0>]}|548]],<br> [],<br> [[{{supervisor,connector_sup,1},[connector,<0.42.0>]}|3],<br>
[{{connector_serv,init,1},[connector_supssl,connector,<0.42.0>]}|1460],<br> [{{supervisor,casserl_sup,1},[connector,<0.42.0>]}|1],<br> [{{supervisor,pushiphone_sup,1},[connector,<0.42.0>]}|2],<br>
[{{pushiphone,init,1},['pushiphone-lite',connector,<0.42.0>]}|3],<br> [{{supervisor,clientpool_sup,1},[connector,<0.42.0>]}|1]],<br> [],<br> [[{{clientpool,init,1},[clientpool_sup,connector,<0.42.0>]}|1],<br>
[undefined|4]],<br> [],<br> [[{{supervisor,connector,1},[<0.42.0>]}|1],<br> [{{casserl_serv,init,1},[casserl_sup,connector,<0.42.0>]}|50]],<br> [],[],[],<br>
[[{{connector_serv,init,1},[connector_suprc4,connector,<0.42.0>]}|472],<br> [{{ssl_connection,init,1},<br> [ssl_connection_sup,ssl_sup,<0.51.0>]}|<br> 1366],<br>
[{unknown,unknown}|3]],<br> [],[],<br> [[{{pushiphone,init,1},['pushiphone-full',connector,<0.42.0>]}|3]],<br> [],<br> [[{{pg2,init,1},[kernel_safe_sup,kernel_sup,<0.10.0>]}|1]]}}}<br>
ok</div><div><br></div><div>I'm very satisfied with supervisor, and I don't think to have the expertise tweaking it...</div><br><div><div>Le 23 mai 2013 à 14:19, Dmitry Kolesnikov <<a href="mailto:dmkolesnikov@gmail.com" target="_blank">dmkolesnikov@gmail.com</a>> a écrit :</div>
<br><blockquote type="cite"><div style="word-wrap:break-word"><br><div><div>On May 23, 2013, at 1:04 PM, Morgan Segalis <<a href="mailto:msegalis@gmail.com" target="_blank">msegalis@gmail.com</a>> wrote:</div><br><blockquote type="cite">
<span style="font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none">I have made a little function a while back, getting all processes and removing the processes inited at the beginning…</span></blockquote>
</div><br><div>Could you please elaborate on that? Why you are not satisfied with supervisor?</div><div><br></div><div>- Dmitry </div></div></blockquote></div><br></div></blockquote></div><br></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
<br></blockquote></div>
</blockquote></div><br></div></div></blockquote></div><br></div></blockquote></div><br></div></blockquote></div><br></body></html>