[erlang-questions] beam.smp high memory usage

[Frank Muller]

Hi guys,

I would like to thank you for your feedbacks and pointers.

After days of unsuccessful investigations, Jesper’s hint led me directly
the culprit.

All clients were sending small binaries to my app, but one. This one guy
sent very large binaries for very long time (keep-alive).

A temporary workaround was to simple limit the size of each binary sent and
force closing TCP connection after N seconds.

The memory usage went immediately back to ~160MB.

Thanks again
/Frank

Fred's excellent "Erlang In Anger" book (https://www.erlang-in-anger.com/)
> has a section on hunting down binary leaks (if that is what you are working
> with here). At least this can give you some starting information about what
> to look for and how to go about handling it afterwards. Identifying which
> processes are holding the binary data by extension identifies the code
> modules which are culprits.
>
> Since your normal operation uses 160 megabytes of memory and it suddenly
> shoots to gigabytes, one of the parts is to identify what is being done to
> the system when that happens. Perhaps you have an outside user which uses
> your system in a way that wasn't thought about originally. This can lead to
> you handling different data and in turn this can expose some of these
> problems.
>
> On Mon, Nov 13, 2017 at 9:23 AM Frank Muller <frank.muller.erl@REDACTED>
> wrote:
>
>> Hi Jesper,
>>
>> Thanks again for the feedback.
>>
>> But from where should I start (Erlang newbie here)?
>> I’ve no idea ... is there a guideline to follow?
>>
>> I feel guessing without making any real progress for the last two days :-/
>>
>> /Frank
>>
>> First quick stab:
>>>
>>> Most of the allocated space is binary() data.
>>>
>>> Typical scenario: something is keeping the binary() data alive, one way
>>> or the other. Either because you need a binary:copy/1 in key locations due
>>> to sub-binaries, or because your system can't get rid of the data since you
>>> still have a pointer kept to it from some process.
>>>
>>>
>>> On Sun, Nov 12, 2017 at 12:51 PM Frank Muller <
>>> frank.muller.erl@REDACTED> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I’ve a fairly complex I/O bound Erlang application.
>>>>
>>>> Since yesterday, the beam.smp memory usage dramatically increased from
>>>> ~160MB to something between [2.5GB,  3.1GB].
>>>> I’ve no idea how, nor why :-/
>>>>
>>>> Attached is snapshot taken with recon_alloc:snapshot().
>>>> Please let me know if you want anything else.
>>>>
>>>> Hope someone can explain me what’s going on.
>>>>
>>>> Thank you.
>>>> /Frank
>>>>
>>>> INFO:
>>>> . Physical machine (not a VM)
>>>> . Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10]
>>>> [async-threads:10] [hipe] [kernel-poll:false]
>>>> . CentOS-7-x86_64
>>>> . Linux ns342284 4.9.58-xxxx-std-ipv6-64 #1 SMP Mon Oct 23 11:35:59
>>>> CEST 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>> . 8x Intel(R) Xeon(R) CPU  E5504  @ 2.00GHz
>>>> . 16GB of RAM
>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20171115/3f299e8e/attachment.htm>