[erlang-questions] Any guidance on tweaking allocator settings?

[Jordan Day]

I’m facing an issue with a program that is using hackney to download a number of large-ish (~8MB) JSON responses. OS-reported memory consumption is typically much larger than I’d expect (I’ve seen spikes of several GB), so I’ve spent quite a bit of time digging through the mailing list and blog posts trying to gain some insight into my problem. There have been several threads discussing the “large binary” problem, which seems like it might be what I’m facing, but a lot of the suggested remedies (lowering fullsweep_after, manually running garbage_collect/1) haven’t seemed to have much effect. After watching Lukas Larsson’s talk from EUC (https://www.youtube.com/watch?v=YuPaX11vZyI), I’m wondering about tweaking allocator settings, but I’m really not sure where to start? While the problem I’m facing seems a lot like the “large binary problem”, observer is reporting that most of my memory usage is tied up in eheap_alloc (https://imgur.com/OzPLvF0). I’ve taken some samples of the memory info over time using recon’s snapshot tool, which can be seen here:
https://gist.github.com/jordan0day/3a127046514a5a9efb87665223e985c7
https://gist.github.com/jordan0day/679e5cc546cdfb4b8c7431f83305b86f
https://gist.github.com/jordan0day/b2f6e69733732f35b612ab810a330ea5

These samples were taken just a minute or two apart from each other, spanning about 30 requests total, so I’d expect the growth to be around 240 MB or so, but you can see by the 3rd snapshot total memory usage is around 1.3 GB.

Everything I’ve read so far has basically said “don’t mess with the allocator settings unless you know what you’re doing!” — and I definitely don’t know what I’m doing, but I’m really hoping to learn! I’d appreciate any guidance anyone can offer.

The information contained in this message is confidential information and privileged. It is intended only for the use of the individual or individuals or entity identified above. You are hereby notified that any dissemination, distribution, use or copying of this message is strictly prohibited.

If the receiver of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of this message is strictly prohibited. If you have received this message in error, please immediately notify the sender by replying to his/her e-mail address noted above and delete the message. Thank You

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160404/ce05ffee/attachment.htm>