[erlang-questions] How can I release beam process memory?
Dániel Szoboszlay
dszoboszlay@REDACTED
Thu Apr 6 13:52:52 CEST 2017
Hi Jack,
*tl;dr;* try starting Erlang with '+MEas bf'
Long explanation:
If the memory is used by the ets_alloc allocator, it can be either actively
used by ETS tables (such as Mnesia disc_copies tables) or it can be lost
due to memory fragmentation. I guess this is memory fragmentation in your
case, but let's quickly rule out the other possibility first. (By the way,
in either case, garbage collecting processes won't reclaim the memory. GC
operates on eheap_alloc allocated memory.)
So if erlang:memory(ets) is close to what recon reports as allocated, the
memory is in use. Your system wrote a lot of data to ETS/Mnesia, and didn't
remove it afterwards. Inspect the contents of the tables and figure out
what was left there that shouldn't be there, which part of the application
should have removed that data and why didn't it do its job properly.
The other, more likely option is fragmentation. I also experienced that the
default memory allocation strategy (aoffcbf = address order first fit
carrier best fit) can perform poorly when you use a lot of memory. All
address order first fit strategies will use heavily the multiblock carriers
with the lowest memory addresses, and if you have many carriers, those
placed higher in memory will have less and less chance to be used. In my
particular case ets_alloc created a total of 150k multiblock carriers for
storing ~1.2TB data in ETS tables. This resulted in ~100 GB unused memory
being wasted in the high address carriers. You have a much smaller system,
but depending on the usage patterns of your ETS data you can end up in a
similar situation.
You can check the number of carriers with erlang:system_info({allocator,
ets_alloc}). It will print something like this:
[{instance,0,
[{versions,"0.9","3.0"},
{options,[...]},
{mbcs,[...
{carriers,1,1,1}, %% <- number of multi block carriers
= 1
...]},
{sbcs,[...
{carriers,0,0,0}, %% <- number of single block carriers
= 0
...]},
{calls,[...]}]},
{instance,1,...
Check theaw numbera across all your allocator instances. You will typically
have very few single block carriers (unless you store huge records in ETS).
In my experience, fragmentation correlates well with the number of carriers
an allocator handles, and can become quite significant above ~10 carriers.
So, If you have the same problem I described, I have some bad news and some
good news. The bad news is that I don't know a way of forcing the VM to
defragment memory and get rid of the waste. The good news is that the bf
(best fit) allocation strategy (which used to be the default up to R16)
performs much better when you have many carriers. You need to pass the '+MEas
bf' command line argument to the VM to switch ets_alloc to bf strategy.
Hope it helps,
Daniel
On Sat, 1 Apr 2017 at 05:36 Jack Tang <himars@REDACTED> wrote:
> After setting up erlang memory visualization, we find etc allocator does
> not release the memory during some historical datum are remove from mnesia
> tables. Can I release the memory on the fly rather than restart the mnesia
> application? Thanks!
>
>
>
>
> BR
>
> On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <dszoboszlay@REDACTED>
> wrote:
>
> Hi Jack,
>
> I guess the 9 GB is lost due to memory fragmentation. Erlang allocates
> memory in large chunks called carriers from the OS, then places the blocks
> your program actually needs on these carriers. A carrier can only be
> returned to the OS once all the blocks on it have been freed (and even
> then, the memory allocator may decide to keep it around for a while in case
> more memory is needed).
>
> You can check with recon_alloc
> <https://ferd.github.io/recon/recon_alloc.html> how much unused memory is
> lost due to fragmentation in the various allocators.
>
> The bad news is that you cannot defragment the carriers, and if the
> selected memory allocator strategy doesn't work well for your application,
> you cannot change it either without restarting the emulator.
>
> However, if the memory is wasted in the eheap_alloc, you may try to force
> a GC on all processes a couple of times. As the GC copies the memory, it
> will allocate new blocks and free up the old heap blocks. So there's a
> chance the allocators can compact the blocks together on fewer segments.
> But that's just a guess, it may or may not work at all.
>
> Cheers,
> Daniel
>
> On Sat, 14 Jan 2017 at 08:04 Jack Tang <himars@REDACTED> wrote:
>
> Hello list,
>
> I run one Erlang application on Debian server and today I find the beam
> process consumes around 35G memory by `top` command.
>
> ```
> KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers
> KiB Swap: 0 total, 0 used, 0 free. 18684864 cached Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp
> ```
>
> I connect to the Erlang application using remote shell and find the
> mem-leaked supervisor tree and run gc on the whole tree. Code looks like
> blow:
>
> ```
> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId)
> end, supervisor:which_children(some_thing_sup)).
> ```
>
> and erlang:memory() decreases from 32G to 23G.
> ```
> [{total,22982011544},
> {processes,12884182336},
> {processes_used,12884170336},
> {system,10097829208},
> {atom,13828705},
> {atom_used,13796692},
> {binary,170530288},
> {code,16450626},
> {ets,9637717576}]
> ```
>
> However, when I input `top` command, the beam process still takes 35G
> memory. What can I do to release the 9G memory? Thanks
>
> BR
> -Jack
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --
> Jack Tang
>
>
> http://www.linkedin.com/in/jacktang
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170406/9b4a4303/attachment.htm>
More information about the erlang-questions
mailing list