<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I’m facing an issue with a program that is using hackney to download a number of large-ish (~8MB) JSON responses. OS-reported memory consumption is typically much larger than I’d expect (I’ve seen spikes of several GB), so I’ve spent quite a bit of time digging through the mailing list and blog posts trying to gain some insight into my problem. There have been several threads discussing the “large binary” problem, which seems like it might be what I’m facing, but a lot of the suggested remedies (lowering fullsweep_after, manually running garbage_collect/1) haven’t seemed to have much effect. After watching Lukas Larsson’s talk from EUC (<a href="https://www.youtube.com/watch?v=YuPaX11vZyI" class="">https://www.youtube.com/watch?v=YuPaX11vZyI</a>), I’m wondering about tweaking allocator settings, but I’m really not sure where to start? While the problem I’m facing seems a lot like the “large binary problem”, observer is reporting that most of my memory usage is tied up in eheap_alloc (<a href="https://imgur.com/OzPLvF0" class="">https://imgur.com/OzPLvF0</a>). I’ve taken some samples of the memory info over time using recon’s snapshot tool, which can be seen here:<div class=""><a href="https://gist.github.com/jordan0day/3a127046514a5a9efb87665223e985c7" class="">https://gist.github.com/jordan0day/3a127046514a5a9efb87665223e985c7</a></div><div class=""><a href="https://gist.github.com/jordan0day/679e5cc546cdfb4b8c7431f83305b86f" class="">https://gist.github.com/jordan0day/679e5cc546cdfb4b8c7431f83305b86f</a></div><div class=""><a href="https://gist.github.com/jordan0day/b2f6e69733732f35b612ab810a330ea5" class="">https://gist.github.com/jordan0day/b2f6e69733732f35b612ab810a330ea5</a></div><div class=""><br class=""></div><div class="">These samples were taken just a minute or two apart from each other, spanning about 30 requests total, so I’d expect the growth to be around 240 MB or so, but you can see by the 3rd snapshot total memory usage is around 1.3 GB.</div><div class=""><br class=""></div><div class="">Everything I’ve read so far has basically said “don’t mess with the allocator settings unless you know what you’re doing!” — and I definitely don’t know what I’m doing, but I’m really hoping to learn! I’d appreciate any guidance anyone can offer.</div><br>The information contained in this message is confidential information and privileged. It is intended only for the use of the individual or individuals or entity identified above. You are hereby notified that any dissemination, distribution, use or copying of this message is strictly prohibited.
<BR><BR>
If the receiver of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, use or copying of this message is strictly prohibited. If you have received this message in error, please immediately notify the sender by replying to his/her e-mail address noted above and delete the message. Thank You
</body></html>