[erlang-questions] Investigate an infinite loop on production servers

[Dmitry Kolesnikov]

You system definitely leaking some resources :-/
 - Check number of used FD(s) may be you exceeded limit there 
 - What was overall system memory / cpu utilisation before crash?
 - Check how many connections you got before crash, may be you can reproduce it at dev

- Dmitry

On May 24, 2013, at 12:13 AM, Morgan Segalis <msegalis@REDACTED> wrote:

> Ok, it finally got into the infinite loop…
> 
> And of course, the node on which I was running etop could not give me anymore since it got disconnected from the production node.
> 
> So back to square one… no way to investigate correctly so far :-/
> 
> Morgan.
> 
> Le 23 mai 2013 à 16:34, Morgan Segalis <msegalis@REDACTED> a écrit :
> 
>> Yeah that what I'm doing right now, but of course, when I'm monitoring it, it won't crash, only when I sleep !!
>> 
>> I get back to the Erlang list as soon as I have more informations about this.
>> 
>> Thank you all !
>> 
>> Morgan.
>> 
>> Le 23 mai 2013 à 16:30, Vance Shipley <vances@REDACTED> a écrit :
>> 
>>> Keep etop running and capture the output to a file (e.g. etop ... | tee stop.log). After it gets into trouble look back and see what was happening beforehand.
>>> On May 23, 2013 6:16 PM, "Morgan Segalis" <msegalis@REDACTED> wrote:
>>> So I should go back to R15B ?
>>> 
>>> erlang:memory() gives me 
>>> 
>>> [{total,1525779584},
>>>  {processes,1272881427},
>>>  {processes_used,1272789743},
>>>  {system,252898157},
>>>  {atom,372217},
>>>  {atom_used,346096},
>>>  {binary,148093608},
>>>  {code,8274446},
>>>  {ets,1546832}]
>>> 
>>> 
>>> But keep in mind that right now, there is no infinite loop, or memory issue at this exact time…
>>> It will be more interesting to have that when the VM is asking for 14GB of memory, but when it does, the console is unresponsive, so I can't get anything then.
>>> 
>>> Le 23 mai 2013 à 14:39, Dmitry Kolesnikov <dmkolesnikov@REDACTED> a écrit :
>>> 
>>>> Right, you do not have many processes. Same time you goes out of memory…
>>>> 
>>>> Unfortunately, I had no time play around with R16B at production… 
>>>> Could it be some issue with SSL, I re-call there was some complains in the list? 
>>>> 
>>>> I would use entop to spot the process that has either too much reductions, queue len or heap.
>>>> Once you know they pid you can dig more info about them using erlang:process_info(…) and/or sys:get:status(…)
>>>> 
>>>> BTW, What erlang:memory() says on you production node?
>>>> 
>>>> - Dmitry
>>>> 
>>>> On May 23, 2013, at 3:25 PM, Morgan Segalis <msegalis@REDACTED> wrote:
>>>> 
>>>>> No, I was talking about the function I made to investigate which processes I have created, which gives me this output : 
>>>>> 
>>>>> Dict: {dict,16,16,16,8,80,48,
>>>>>            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
>>>>>            {{[[{{connector_serv,init,1},[connector_suprc42,connector,<0.42.0>]}|548]],
>>>>>              [],
>>>>>              [[{{supervisor,connector_sup,1},[connector,<0.42.0>]}|3],
>>>>>               [{{connector_serv,init,1},[connector_supssl,connector,<0.42.0>]}|1460],
>>>>>               [{{supervisor,casserl_sup,1},[connector,<0.42.0>]}|1],
>>>>>               [{{supervisor,pushiphone_sup,1},[connector,<0.42.0>]}|2],
>>>>>               [{{pushiphone,init,1},['pushiphone-lite',connector,<0.42.0>]}|3],
>>>>>               [{{supervisor,clientpool_sup,1},[connector,<0.42.0>]}|1]],
>>>>>              [],
>>>>>              [[{{clientpool,init,1},[clientpool_sup,connector,<0.42.0>]}|1],
>>>>>               [undefined|4]],
>>>>>              [],
>>>>>              [[{{supervisor,connector,1},[<0.42.0>]}|1],
>>>>>               [{{casserl_serv,init,1},[casserl_sup,connector,<0.42.0>]}|50]],
>>>>>              [],[],[],
>>>>>              [[{{connector_serv,init,1},[connector_suprc4,connector,<0.42.0>]}|472],
>>>>>               [{{ssl_connection,init,1},
>>>>>                 [ssl_connection_sup,ssl_sup,<0.51.0>]}|
>>>>>                1366],
>>>>>               [{unknown,unknown}|3]],
>>>>>              [],[],
>>>>>              [[{{pushiphone,init,1},['pushiphone-full',connector,<0.42.0>]}|3]],
>>>>>              [],
>>>>>              [[{{pg2,init,1},[kernel_safe_sup,kernel_sup,<0.10.0>]}|1]]}}}
>>>>> ok
>>>>> 
>>>>> I'm very satisfied with supervisor, and I don't think to have the expertise tweaking it...
>>>>> 
>>>>> Le 23 mai 2013 à 14:19, Dmitry Kolesnikov <dmkolesnikov@REDACTED> a écrit :
>>>>> 
>>>>>> 
>>>>>> On May 23, 2013, at 1:04 PM, Morgan Segalis <msegalis@REDACTED> wrote:
>>>>>> 
>>>>>>> I have made a little function a while back, getting all processes and removing the processes inited at the beginning…
>>>>>> 
>>>>>> Could you please elaborate on that? Why you are not satisfied with supervisor?
>>>>>> 
>>>>>> - Dmitry 
>>>>> 
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130524/2721463d/attachment.htm>