[erlang-questions] How can I release beam process memory?
Jack Tang
himars@REDACTED
Thu Apr 6 16:41:42 CEST 2017
Hi Daniel
Thank you for the detailed explanation and sharing the experiences. I
focused on the performance of GC heavily before and ignored the allocation
strategy.
One more question, as you mentioned memory on eheap_alloc can be released
by calling GC on every process. Right now I setup periodic gc process to
free eheap. However can I set some process flag to make GC automatically?
Thanks again!
BR
-Jack
On Thu, Apr 6, 2017 at 7:52 PM, Dániel Szoboszlay <dszoboszlay@REDACTED>
wrote:
> Hi Jack,
>
> *tl;dr;* try starting Erlang with '+MEas bf'
>
> Long explanation:
>
> If the memory is used by the ets_alloc allocator, it can be either
> actively used by ETS tables (such as Mnesia disc_copies tables) or it can
> be lost due to memory fragmentation. I guess this is memory fragmentation
> in your case, but let's quickly rule out the other possibility first. (By
> the way, in either case, garbage collecting processes won't reclaim the
> memory. GC operates on eheap_alloc allocated memory.)
>
> So if erlang:memory(ets) is close to what recon reports as allocated, the
> memory is in use. Your system wrote a lot of data to ETS/Mnesia, and didn't
> remove it afterwards. Inspect the contents of the tables and figure out
> what was left there that shouldn't be there, which part of the application
> should have removed that data and why didn't it do its job properly.
>
> The other, more likely option is fragmentation. I also experienced that
> the default memory allocation strategy (aoffcbf = address order first fit
> carrier best fit) can perform poorly when you use a lot of memory. All
> address order first fit strategies will use heavily the multiblock carriers
> with the lowest memory addresses, and if you have many carriers, those
> placed higher in memory will have less and less chance to be used. In my
> particular case ets_alloc created a total of 150k multiblock carriers for
> storing ~1.2TB data in ETS tables. This resulted in ~100 GB unused memory
> being wasted in the high address carriers. You have a much smaller system,
> but depending on the usage patterns of your ETS data you can end up in a
> similar situation.
>
> You can check the number of carriers with erlang:system_info({allocator,
> ets_alloc}). It will print something like this:
> [{instance,0,
> [{versions,"0.9","3.0"},
> {options,[...]},
> {mbcs,[...
> {carriers,1,1,1}, %% <- number of multi block
> carriers = 1
> ...]},
> {sbcs,[...
> {carriers,0,0,0}, %% <- number of single block
> carriers = 0
> ...]},
> {calls,[...]}]},
> {instance,1,...
> Check theaw numbera across all your allocator instances. You will
> typically have very few single block carriers (unless you store huge
> records in ETS). In my experience, fragmentation correlates well with the
> number of carriers an allocator handles, and can become quite significant
> above ~10 carriers.
>
> So, If you have the same problem I described, I have some bad news and
> some good news. The bad news is that I don't know a way of forcing the VM
> to defragment memory and get rid of the waste. The good news is that the bf
> (best fit) allocation strategy (which used to be the default up to R16)
> performs much better when you have many carriers. You need to pass the '+MEas
> bf' command line argument to the VM to switch ets_alloc to bf strategy.
>
> Hope it helps,
> Daniel
>
> On Sat, 1 Apr 2017 at 05:36 Jack Tang <himars@REDACTED> wrote:
>
>> After setting up erlang memory visualization, we find etc allocator does
>> not release the memory during some historical datum are remove from mnesia
>> tables. Can I release the memory on the fly rather than restart the mnesia
>> application? Thanks!
>>
>>
>>
>>
>> BR
>>
>> On Sun, Jan 15, 2017 at 4:47 AM, Dániel Szoboszlay <dszoboszlay@REDACTED
>> > wrote:
>>
>> Hi Jack,
>>
>> I guess the 9 GB is lost due to memory fragmentation. Erlang allocates
>> memory in large chunks called carriers from the OS, then places the blocks
>> your program actually needs on these carriers. A carrier can only be
>> returned to the OS once all the blocks on it have been freed (and even
>> then, the memory allocator may decide to keep it around for a while in case
>> more memory is needed).
>>
>> You can check with recon_alloc
>> <https://ferd.github.io/recon/recon_alloc.html> how much unused memory
>> is lost due to fragmentation in the various allocators.
>>
>> The bad news is that you cannot defragment the carriers, and if the
>> selected memory allocator strategy doesn't work well for your application,
>> you cannot change it either without restarting the emulator.
>>
>> However, if the memory is wasted in the eheap_alloc, you may try to
>> force a GC on all processes a couple of times. As the GC copies the memory,
>> it will allocate new blocks and free up the old heap blocks. So there's a
>> chance the allocators can compact the blocks together on fewer segments.
>> But that's just a guess, it may or may not work at all.
>>
>> Cheers,
>> Daniel
>>
>> On Sat, 14 Jan 2017 at 08:04 Jack Tang <himars@REDACTED> wrote:
>>
>> Hello list,
>>
>> I run one Erlang application on Debian server and today I find the beam
>> process consumes around 35G memory by `top` command.
>>
>> ```
>> KiB Mem: 99194912 total, 61682656 used, 37512252 free, 397380 buffers
>> KiB Swap: 0 total, 0 used, 0 free. 18684864 cached
>> Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 11858 usr1 20 0 36.850g 0.032t 6220 S 73.5 34.4 8038:49 beam.smp
>> ```
>>
>> I connect to the Erlang application using remote shell and find the
>> mem-leaked supervisor tree and run gc on the whole tree. Code looks like
>> blow:
>>
>> ```
>> lists:foreach(fun(E) -> PId = element(2, E), erlang:garbage_collect(PId)
>> end, supervisor:which_children(some_thing_sup)).
>> ```
>>
>> and erlang:memory() decreases from 32G to 23G.
>> ```
>> [{total,22982011544},
>> {processes,12884182336},
>> {processes_used,12884170336},
>> {system,10097829208},
>> {atom,13828705},
>> {atom_used,13796692},
>> {binary,170530288},
>> {code,16450626},
>> {ets,9637717576}]
>> ```
>>
>> However, when I input `top` command, the beam process still takes 35G
>> memory. What can I do to release the 9G memory? Thanks
>>
>> BR
>> -Jack
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>> --
>> Jack Tang
>>
>>
>> http://www.linkedin.com/in/jacktang
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
--
Jack Tang
http://www.linkedin.com/in/jacktang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20170406/ba5cea2e/attachment.htm>
More information about the erlang-questions
mailing list