[erlang-questions] VM leaking memory

[Frank Muller]

Thanks Fred. Your recon library helped me a lot with additional tools like
observer_cli (which wraps most of recon calls in a nice console GUI):
https://github.com/zhongwencool/observer_cli/blob/master/README.md

I’ve most of these information in my possession as I said to share in
private. I will try the other ideas and report back.

/Frank

On 01/31, Frank Muller wrote:
> >After adding a new feature to my app (running non-stop for 5 years), it
> >started leaking memory in staging.
> >
> >Obviously, I’m suspecting this new feature. Command top shows RES going
> >from 410m (during startup) to 6.2g in less than 12h.
> >
> >For stupid security reasons, it will take me weeks to be allowed to share
> >collected statistics (from recon, entop) here, but I can share them in
> >private if someone is willing to help.
> >
>
> I'd recommend checking things like:
>
> - recon_alloc:memory(usage) and see if the ratio is high or very low;
>   this can point towards memory fragmentation if the ratio is low.
> - in case there is fragmentation (or something that looks like it)
>   recon_alloc:fragmentation(current) will return lists of all the
>   various allocators and types, which should help point towards which
>   type of memory is causing issues
> - if usage seems high, see recon_alloc:memory(allocated_types) to see if
>   there's any allocator that's higher than others; ETS, binary, or eheap
>   will tend to point towards an ETS table, a refc binary leak, or some
>   process gathering lots of memory
>
> Based on this it might be possible to then orient towards other avenues
> without you having to share any numbers.
>
> Quick checks if it's binary memory is to call recon:bin_leak(10), which
> will probe all processes for their binary memory usage, run a GC on all
> of them, then run a probe again, and give you those that have the
> largest gap. This can point to processes that had the most dead memory.
>
> There's an undocumented 'binary_memory' option that recon:info,
> recon:proc_count, and recon:proc_window all support -- it's undocumented
> because it might be expensive and not always safe to run -- that you can
> use to find which processes are holding the most binary memory; after a
> call to bin_leak, this can let you know about biggest users.
>
> You can also use proc_count with:
> - message_queue_len for large mailboxes
> - memory for eheap usage
>
> You can use the same values with proc_window to see who is currently
> allocating the most.
>
> If ETS is taking a lot of place, calling ets:i() can show a bunch of
> tables with content; you might have a runaway cache table or something
> like that.
>
> Regards,
> Fred.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190131/be42ffaf/attachment.htm>