[erlang-questions] Any guidance on tweaking allocator settings?

[Lukas Larsson]

Hello,

If you use recon_alloc:memory(usage) on the snapshots you have you can see
you have a high usage percentage which means that you do not have any
memory fragmentation issues that tweaking the erts allocators would fix.
Your application is simply using a lot of memory and it seems to be process
heap memory. I would recommend using observer to see which processes use a
lot of memory and from there try to figure out why.

Lukas

On Mon, Apr 4, 2016 at 11:53 PM, Jordan Day <jordand@REDACTED>
wrote:

> I’m facing an issue with a program that is using hackney to download a
> number of large-ish (~8MB) JSON responses. OS-reported memory consumption
> is typically much larger than I’d expect (I’ve seen spikes of several GB),
> so I’ve spent quite a bit of time digging through the mailing list and blog
> posts trying to gain some insight into my problem. There have been several
> threads discussing the “large binary” problem, which seems like it might be
> what I’m facing, but a lot of the suggested remedies (lowering
> fullsweep_after, manually running garbage_collect/1) haven’t seemed to have
> much effect. After watching Lukas Larsson’s talk from EUC (
> https://www.youtube.com/watch?v=YuPaX11vZyI), I’m wondering about
> tweaking allocator settings, but I’m really not sure where to start? While
> the problem I’m facing seems a lot like the “large binary problem”,
> observer is reporting that most of my memory usage is tied up in
> eheap_alloc (https://imgur.com/OzPLvF0). I’ve taken some samples of the
> memory info over time using recon’s snapshot tool, which can be seen here:
> https://gist.github.com/jordan0day/3a127046514a5a9efb87665223e985c7
> https://gist.github.com/jordan0day/679e5cc546cdfb4b8c7431f83305b86f
> https://gist.github.com/jordan0day/b2f6e69733732f35b612ab810a330ea5
>
> These samples were taken just a minute or two apart from each other,
> spanning about 30 requests total, so I’d expect the growth to be around 240
> MB or so, but you can see by the 3rd snapshot total memory usage is around
> 1.3 GB.
>
> Everything I’ve read so far has basically said “don’t mess with the
> allocator settings unless you know what you’re doing!” — and I definitely
> don’t know what I’m doing, but I’m really hoping to learn! I’d appreciate
> any guidance anyone can offer.
>
> The information contained in this message is confidential information and
> privileged. It is intended only for the use of the individual or
> individuals or entity identified above. You are hereby notified that any
> dissemination, distribution, use or copying of this message is strictly
> prohibited.
>
> If the receiver of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution, use or copying of
> this message is strictly prohibited. If you have received this message in
> error, please immediately notify the sender by replying to his/her e-mail
> address noted above and delete the message. Thank You   ­­
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160405/02ec577c/attachment.htm>