[erlang-questions] Binary leak on VM? How to resolve this issue?

[Bogdan Andu]

Hi,

Most likely you have temporary  processes that handles binaries that never 
die
holding those binaries referenced and gc cannnot garbage collect them.

This is a sign o bad process cleanup management logic.

On Tuesday, November 10, 2015 at 4:59:27 AM UTC+2, 이기탁 wrote:
>
> Hello All :)
>
>
> I have a some question about Erlang binary memory management.
>
> I'm running the application on Erlang VM 17.4
> (Erlang/OTP 17 [erts-6.3] [source-f9282c6] [64-bit] [smp:24:24] 
> [async-threads:10] [hipe] [kernel-poll:false])
>
> This application is processing many binaries. (relaying some binary 
> packets from A client to B client)
>
> But A weeks ago, binary metric in system does not drop on base line (like 
> below picture). 
> Base line of binary memory is about 144 MB but only increased :( 
>
>
>
>
> <https://lh3.googleusercontent.com/-1a_Y6_fLALQ/VkFXvIF_i7I/AAAAAAAAAOg/JHjuVsqqTDk/s1600/is_memory_leak.png>
> So I ran some codes in machine's console
>
> (xxx@REDACTED)1> lists:sum([try
> (xxx@REDACTED)1>   {_,M} = erlang:process_info(Pid, binary),
> (xxx@REDACTED)1>   lists:sum([ByteSize || {_, ByteSize, _} <- M])
> (xxx@REDACTED)1> catch
> (xxx@REDACTED)1>   _:_ -> 0
> (xxx@REDACTED)1> end || Pid <- processes()]).
> 151431354
> (xxx@REDACTED)2> CS = lists:foldl(
> (xxx@REDACTED)2>        fun ({instance, _, L}, Acc) ->
> (xxx@REDACTED)2>          {value,{_,MBCS}} = lists:keysearch(mbcs, 1, L),
> (xxx@REDACTED)2>          {value,{_,SBCS}} = lists:keysearch(sbcs, 1, L),
> (xxx@REDACTED)2>          [MBCS,SBCS | Acc]
> (xxx@REDACTED)2>        end,
> (xxx@REDACTED)2>        [],
> (xxx@REDACTED)2>        erlang:system_info({allocator_sizes, binary_alloc})),
> (xxx@REDACTED)2> lists:foldl(
> (xxx@REDACTED)2>   fun(L, Sz0) ->
> (xxx@REDACTED)2>     {value,{_,Sz,_,_}} = lists:keysearch(blocks_size, 1, L),
> (xxx@REDACTED)2>     Sz0+Sz
> (xxx@REDACTED)2>   end, 0, CS).
> 553821984
>
> Sum of binaries using processes is 144MB, but some of binary allocator is 
> 528MB. 
> This is almost same with below binary memory.
>
> (xxx@REDACTED)3> erlang:memory().
> [{total,2314432736},
>  {processes,1672464336},
>  {processes_used,1672334920},
>  {system,641968400},
>  {atom,2017513},
>  {atom_used,2001044},
>  {binary,553927480}, <--
>  {code,54190392},
>  {ets,7211864}]
>
> Maybe GC problem? so I ran GC forcibly.
>
> (xxx@REDACTED)4> [erlang:garbage_collect(P) || P <- processes()].
> [true,true,true,true,true,true,true,true,true,true,true,
>  true,true,true,true,true,true,true,true,true,true,true,true,
>  true,true,true,true,true,true|...]
>
>
> (xxx@REDACTED)5> erlang:memory().
> [{total,2294793888},
>  {processes,1654106832},
>  {processes_used,1653947192},
>  {system,640687056},
>  {atom,2017513},
>  {atom_used,2001044},
>  {binary,552605080}, <--
>  {code,54190392},
>  {ets,7240816}]
>
> Not resolved... :(
>
> Moreover this issue is only in production environment, no issue in develop 
> env. I can't reproduce it.
> About 200 machines in production, but about 110 machines has this issue.
>
> How can I approach this issue to resolve? 
>
> Ah! my sys.config is like this.
> ===
> +K true
> +A 6
> +stbt db
> +SP 50:50
> +P 134217727
> -env ERL_FULLSWEEP_AFTER 0
> -heart
> ===
>
> Thanks in advance for your advice,
>
> GiTack, Lee
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151209/2e14bf53/attachment.htm>