[erlang-questions] Hidden binaries

[Robert Raschke]

Hi Kirill,

is there any chance you are using binary:split to smash a larger binary
into pieces for further processing, thinking that the big thing will go
away asap?

If yes, then I can recommend reading
https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole ,
especially from section "I Just Keep On Bleeding and I Won't Die" onwards.

Naive summary (reality is a bit more discerning :-) : when you have refs
into a larger binary, the large binary won't get gc'ed, and you need to
jump through some hoops to get rid of it.

HTH,
Robby
On May 28, 2014 8:07 AM, "Kirill Zaborsky" <qrilka@REDACTED> wrote:

> Hello,
> I have recently found some memory "leak" in our application. Leak is in
> quotes because memory load seems to be constant under constant load. But
> the problem is that I see some extra binaries for which there are no signs
> in process_info.
> In production we use our fork of OTP16BR3-1 (including some number of
> httpc fixes) but it looks like that behaviour could be reproduced locally
> on my machine with OTP 17.
> I have tried to get any hints from recon but with not success.
> The problem could be seen from the following Erlang shell lines:
> -------------
> (node@REDACTED)87> recon:bin_leak(20),erlang:memory().
> [{total,30574120},
>  {processes,6765096},
>  {processes_used,6738520},
>  {system,23809024},
>  {atom,662409},
>  {atom_used,652067},
>  {binary,145760},
>  {code,16742599},
>  {ets,1037976}]
> (node@REDACTED)88>
> lists:sum([N||{P1,N,B}<-[{P,lists:sum([S||{_,S,_}<-B]),B} || {P,{binary,B}}
> <- [{Pid,(catch process_info(Pid,binary))} || Pid <- processes()]]]).
> 17671
> (node@REDACTED)89>
> -------------
> As you see erlang:memory/0 shows 146k of memory used by binaries but
> erlang:process_info/2 shows information only about 18k
> Is there any way I could find out where the missing 128k of memory is?
> Is it possible to get all binaries contents without gowing to some
> low-level tools like gdb (I'm not quite familiar with it yet)?
> Possibly there are some system-level binaries?
> In production I see not kilobytes but hunderds of megabytes and sometimes
> it leads even to OOM crashes.
> I run Gentoo on my desktop and in production we use Ubuntu Precise.
> Thanks for any advice.
>
> Best regards,
> Kirill Zaborsky
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140528/d0b1bc40/attachment.htm>