[erlang-questions] How can I release beam process memory?

Jack Tang himars@REDACTED
Thu Apr 6 16:41:42 CEST 2017

Hi Daniel

Thank you for the detailed explanation and sharing the experiences. I
focused on the performance of GC heavily before and ignored the allocation

One more question, as you mentioned memory on eheap_alloc can be released
by calling GC on every process. Right now I setup periodic gc process  to
free eheap. However can I set some process flag to make GC automatically?

Thanks again!


On Thu, Apr 6, 2017 at 7:52 PM, Dániel Szoboszlay <dszoboszlay@REDACTED>

> Hi Jack,
> *tl;dr;* try starting Erlang with '+MEas bf'
> Long explanation:
> If the memory is used by the ets_alloc allocator, it can be either
> actively used by ETS tables (such as Mnesia disc_copies tables) or it can
> be lost due to memory fragmentation. I guess this is memory fragmentation
> in your case, but let's quickly rule out the other possibility first. (By
> the way, in either case, garbage collecting processes won't reclaim the
> memory. GC operates on eheap_alloc allocated memory.)
> So if erlang:memory(ets) is close to what recon reports as allocated, the
> memory is in use. Your system wrote a lot of data to ETS/Mnesia, and didn't
> remove it afterwards. Inspect the contents of the tables and figure out
> what was left there that shouldn't be there, which part of the application
> should have removed that data and why didn't it do its job properly.
> The other, more likely option is fragmentation. I also experienced that
> the default memory allocation strategy (aoffcbf = address order first fit
> carrier best fit) can perform poorly when you use a lot of memory. All
> address order first fit strategies will use heavily the multiblock carriers
> with the lowest memory addresses, and if you have many carriers, those
> placed higher in memory will have less and less chance to be used. In my
> particular case ets_alloc created a total of 150k multiblock carriers for
> storing ~1.2TB data in ETS tables. This resulted in ~100 GB unused memory
> being wasted in the high address carriers. You have a much smaller system,
> but depending on the usage patterns of your ETS data you can end up in a
> similar situation.
> You can check the number of carriers with erlang:system_info({allocator,
> ets_alloc}). It will print something like this:
> [{instance,0,
>            [{versions,"0.9","3.0"},
>             {options,[...]},
>             {mbcs,[...
>                    {carriers,1,1,1},  %% <- number of multi block
> carriers = 1
>                    ...]},
>             {sbcs,[...
>                    {carriers,0,0,0},  %% <- number of single block
> carriers = 0
>                    ...]},
>             {calls,[...]}]},
>  {instance,1,...
> Check theaw numbera across all your allocator instances. You will
> typically have very few single block carriers (unless you store huge
> records in ETS). In my experience, fragmentation correlates well with the
> number of carriers an allocator handles, and can become quite significant
> above ~10 carriers.
> So, If you have the same problem I described, I have some bad news and
> some good news. The bad news is that I don't know a way of forcing the VM
> to defragment memory and get rid of the waste. The good news is that the bf
> (best fit) allocation strategy (which used to be the default up to R16)
> performs much better when you have many carriers. You need to pass the '+MEas
> bf' command line argument to the VM to switch ets_alloc to bf strategy.
> Hope it helps,
> Daniel
> On Sat, 1 Apr 2017 at 05:36 Jack Tang <himars@REDACTED> wrote:
>> After setting up erlang memory visualization, we find etc allocator does
>> not release the memory during some historical datum are remove from mnesia
>> tables. Can I release the memory on the fly rather than restart the mnesia
>> application? Thanks!
>> BR
>> On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <dszoboszlay@REDACTED
>> > wrote:
>> Hi Jack,
>> I guess the 9 GB is lost due to memory fragmentation. Erlang allocates
>> memory in large chunks called carriers from the OS, then places the blocks
>> your program actually needs on these carriers. A carrier can only be
>> returned to the OS once all the blocks on it have been freed (and even
>> then, the memory allocator may decide to keep it around for a while in case
>> more memory is needed).
>> You can check with recon_alloc
>> <https://ferd.github.io/recon/recon_alloc.html> how much unused memory
>> is lost due to fragmentation in the various allocators.
>> The bad news is that you cannot defragment the carriers, and if the
>> selected memory allocator strategy doesn't work well for your application,
>> you cannot change it either without restarting the emulator.
>> However, if the memory is wasted in the eheap_alloc, you may try to
>> force a GC on all processes a couple of times. As the GC copies the memory,
>> it will allocate new blocks and free up the old heap blocks. So there's a
>> chance the allocators can compact the blocks together on fewer segments.
>> But that's just a guess, it may or may not work at all.
>> Cheers,
>> Daniel
>> On Sat, 14 Jan 2017 at 08:04 Jack Tang <himars@REDACTED> wrote:
>> Hello list,
>> I run one Erlang application on Debian server and today I find the beam
>> process consumes around 35G memory by `top` command.
>> ```
>> KiB Mem:  99194912 total, 61682656 used, 37512252 free,   397380 buffers
>> KiB Swap:        0 total,        0 used,        0 free. 18684864 cached
>> Mem
>>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
>> 11858 usr1   20   0 36.850g 0.032t   6220 S  73.5 34.4   8038:49 beam.smp
>> ```
>> I connect to the Erlang application using remote shell and find the
>> mem-leaked supervisor tree and run gc on the whole tree. Code looks like
>> blow:
>> ```
>> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId)
>> end, supervisor:which_children(some_thing_sup)).
>> ```
>> and erlang:memory() decreases from 32G to 23G.
>> ```
>> [{total,22982011544},
>>  {processes,12884182336},
>>  {processes_used,12884170336},
>>  {system,10097829208},
>>  {atom,13828705},
>>  {atom_used,13796692},
>>  {binary,170530288},
>>  {code,16450626},
>>  {ets,9637717576}]
>> ```
>> However, when I input `top` command, the beam process still takes 35G
>> memory. What can I do to release the 9G memory? Thanks
>> BR
>> -Jack
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>> --
>> Jack Tang
>> http://www.linkedin.com/in/jacktang
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions

Jack Tang

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170406/ba5cea2e/attachment.htm>

More information about the erlang-questions mailing list