[erlang-questions] Garbage Collection, BEAM memory and Erlang memory

[Roberto Ostinelli]

Here's the erl_crash.dump analysis, done with the nice Fred's script:

analyzing erl_crash.dump, generated on:  Wed Jan 28 13:59:36 2015

Slogan: Received SIGUSR1

Memory:
===
  processes: 8870 Mb
  processes_used: 8869 Mb
  system: 1138 Mb
  atom: 0 Mb
  atom_used: 0 Mb
  binary: 750 Mb
  code: 9 Mb
  ets: 7 Mb
  ---
  total: 10008 Mb

Different message queue lengths (5 largest different):
===
540314 0

Error logger queue length:
===

File descriptors open:
===
  UDP:  0
  TCP:  180071
  Files:  6
  ---
  Total:  180077

Number of processes:
===
  540314

Processes Heap+Stack memory sizes (words) used in the VM (5 largest
different):
===
   2 196650
   1 28690
   1 17731
   1 10958
4677 6772

Processes OldHeap memory sizes (words) used in the VM (5 largest different):
===
   1 1439468
   1 999631
   1 75113
   1 28690
   1 17731

Process States when crashing (sum):
===
540314 Waiting


Do you see anything wrong there? I honestly don't.

Best,
r.



On Wed, Jan 28, 2015 at 3:15 PM, Roberto Ostinelli <roberto@REDACTED>
wrote:

>
> On Wed, Jan 28, 2015 at 3:08 PM, Anton Lebedevich <mabrek@REDACTED>
> wrote:
>
>> Time scale is different so it's not possible to correlate process
>> memory with number of reductions or number of GCs.
>>
>
> Time scale is different from previous test, but the graph is the same.
> Things get ugly at 140.
>
>
>
>> It seems that some process (or processes) starts allocating memory
>> much faster than before and linux OOM kills the beam when it runs out
>> of memory on the box.
>
>
> How can you see that?
>
>
>
>> You can try setting a watchdog process
>> (something like "while true; check memory usage and kill -USR1
>> beam.smp when it's close to the limit; sleep 1") to get crash dump
>> before OOM kills beam.smp
>>
>
> I did that, I'm currently waiting for the crash dump to finish (it is
> 2.9GB right now, and still piling up).
>
>
>
>> Is there anything unusual in logs at the momeng when memory usage is
>> jumping? Maybe something gets printed to stdout.
>
>
> Nothing unfortunately.
>
> Thank you for your help,
> r.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150128/ce4d9c40/attachment.htm>