[erlang-questions] beam.smp high memory usage

[Jesper Louis Andersen]

Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/)
has a section on hunting down binary leaks (if that is what you are working
with here). At least this can give you some starting information about what
to look for and how to go about handling it afterwards. Identifying which
processes are holding the binary data by extension identifies the code
modules which are culprits.

Since your normal operation uses 160 megabytes of memory and it suddenly
shoots to gigabytes, one of the parts is to identify what is being done to
the system when that happens. Perhaps you have an outside user which uses
your system in a way that wasn't thought about originally. This can lead to
you handling different data and in turn this can expose some of these
problems.

On Mon, Nov 13, 2017 at 9:23 AM Frank Muller <frank.muller.erl@REDACTED>
wrote:

> Hi Jesper,
>
> Thanks again for the feedback.
>
> But from where should I start (Erlang newbie here)?
> I’ve no idea ... is there a guideline to follow?
>
> I feel guessing without making any real progress for the last two days :-/
>
> /Frank
>
> First quick stab:
>>
>> Most of the allocated space is binary() data.
>>
>> Typical scenario: something is keeping the binary() data alive, one way
>> or the other. Either because you need a binary:copy/1 in key locations due
>> to sub-binaries, or because your system can't get rid of the data since you
>> still have a pointer kept to it from some process.
>>
>>
>> On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <frank.muller.erl@REDACTED>
>> wrote:
>>
>>> Hi guys,
>>>
>>> I’ve a fairly complex I/O bound Erlang application.
>>>
>>> Since yesterday, the beam.smp memory usage dramatically increased from
>>> ~160MB to something between [2.5GB,  3.1GB].
>>> I’ve no idea how, nor why :-/
>>>
>>> Attached is snapshot taken with recon_alloc:snapshot().
>>> Please let me know if you want anything else.
>>>
>>> Hope someone can explain me what’s going on.
>>>
>>> Thank you.
>>> /Frank
>>>
>>> INFO:
>>> . Physical machine (not a VM)
>>> . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10]
>>> [async-threads:10] [hipe] [kernel-poll:false]
>>> . CentOS-7-x86_64
>>> . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59 CEST
>>> 2017 x86_64 x86_64 x86_64 GNU/Linux
>>> . 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
>>> . 16GB of RAM
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171113/2ec88eff/attachment.htm>