[erlang-questions] Investigate an infinite loop on production servers

[Bob Ippolito]

This kind of thing tends to happen when you continuously send messages to a
process faster than it can handle them. The most common case that I've seen
this is where you have a lot of processes communicating with a single
gen_server process. If your server has swap enabled, this may appear to
make the node "freeze completely but not crash".

In the past I've diagnosed this by monitoring the message_queue_len of
registered processes, but I'm sure there are tools that can help do this
for you.


On Wed, May 22, 2013 at 7:00 PM, Morgan Segalis <msegalis@REDACTED> wrote:

> Hello everyone,
>
> I'm having a bit of an issue with my production servers.
>
> At some point, it seems to enter into an infinite loop that I can't find,
> or reproduce by myself on the tests servers.
>
> The bug appear completely random, 1 hour, or 10 hour after restarting the
> Erlang node.
> The loop will eat up all my server's memory in no time, and freeze
> completely the Erlang node without crashing it. (most of the time)
>
> One time I got an crash dump, and tried to investigate it with cdv, but I
> didn't get much informations about which process or module was eating up
> all the memory.
> I just know that it crashed because of the crash message : "eheap_alloc:
> Cannot allocate 6801972448 bytes of memory (of type "heap")."
>
> I'm surely too new to Erlang to investigate something like this with cdv,
> I really would like some pointers on how I can understand this problem and
> fix it asap.
>
> If you need any informations about the crash dump, let me know what you
> need, I'll copy/paste…
>
> I'm using Erlang R16B (erts-5.10.1) [source] [64-bit] [smp:8:8]
> [async-threads:10] [kernel-poll:true]
>
> Thank you all for your help !
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130522/186ceb11/attachment.htm>