[erlang-questions] How can I release beam process memory?

[Jack Tang]

Hello,

In the memory issue investigation, I find something interesting. As I
mentioned,
we usually adopted remote Erlang shell to interactive with nodes in
production.
Sometime we use mnesia:select(...) to return some query result, and it may
contain
billions of rows. And we leave the remote shell alive. I find the query
results do
consume the memory of remote node, since the return value is referenced. In
the
shell, we still can access it using `v(N)` command. I tried `f()` command,
and it only
unbinds the variables. So what's the mechanism of Erlang shell gc? Thanks.

BR

On Sun, Jan 15, 2017 at 2:09 PM, Jack Tang <himars@REDACTED> wrote:

> Hi Daniel
>
> Thank you for the clue and excellent recon toolkits! I would like dig it
> deeper  :)
>
> BR
> -Jack
>
> On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <dszoboszlay@REDACTED>
> wrote:
>
>> Hi Jack,
>>
>> I guess the 9 GB is lost due to memory fragmentation. Erlang allocates
>> memory in large chunks called carriers from the OS, then places the blocks
>> your program actually needs on these carriers. A carrier can only be
>> returned to the OS once all the blocks on it have been freed (and even
>> then, the memory allocator may decide to keep it around for a while in case
>> more memory is needed).
>>
>> You can check with recon_alloc
>> <https://ferd.github.io/recon/recon_alloc.html> how much unused memory
>> is lost due to fragmentation in the various allocators.
>>
>> The bad news is that you cannot defragment the carriers, and if the
>> selected memory allocator strategy doesn't work well for your application,
>> you cannot change it either without restarting the emulator.
>>
>> However, if the memory is wasted in the eheap_alloc, you may try to
>> force a GC on all processes a couple of times. As the GC copies the memory,
>> it will allocate new blocks and free up the old heap blocks. So there's a
>> chance the allocators can compact the blocks together on fewer segments.
>> But that's just a guess, it may or may not work at all.
>>
>> Cheers,
>> Daniel
>>
>> On Sat, 14 Jan 2017 at 08:04 Jack Tang <himars@REDACTED> wrote:
>>
>>> Hello list,
>>>
>>> I run one Erlang application on Debian server and today I find the beam
>>> process consumes around 35G memory by `top` command.
>>>
>>> ```
>>> KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
>>> KiB Swap:        0 total,        0 used,        0 free. 18684864 cached
>>> Mem
>>>
>>>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>>> COMMAND
>>> 11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
>>> ```
>>>
>>> I connect to the Erlang application using remote shell and find the
>>> mem-leaked supervisor tree and run gc on the whole tree. Code looks like
>>> blow:
>>>
>>> ```
>>> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId)
>>> end, supervisor:which_children(some_thing_sup)).
>>> ```
>>>
>>> and erlang:memory() decreases from 32G to 23G.
>>> ```
>>> [{total,22982011544},
>>>  {processes,12884182336},
>>>  {processes_used,12884170336},
>>>  {system,10097829208},
>>>  {atom,13828705},
>>>  {atom_used,13796692},
>>>  {binary,170530288},
>>>  {code,16450626},
>>>  {ets,9637717576}]
>>> ```
>>>
>>> However, when I input `top` command, the beam process still takes 35G
>>> memory. What can I do to release the 9G memory? Thanks
>>>
>>> BR
>>> -Jack
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>
>
> --
> Jack Tang
>
>
> http://www.linkedin.com/in/jacktang
>



-- 
Jack Tang


http://www.linkedin.com/in/jacktang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170123/810e7e65/attachment.htm>