[erlang-questions] Problem with Beam process not freeing memory
Michael Truog
mjtruog@REDACTED
Tue Sep 17 17:31:32 CEST 2013
erlang:garbage_collect/0,1 existing is nice, but using it seems to only form a kludge. It seems much better to focus on having short-lived Erlang processes to capture the last reference to generated binaries (that are large enough to be reference counted, so greater than 64 bytes). The pid options for fullsweep, hibernate, erts command line options, and perhaps erlang:garbage_collect/0,1 usage, seems more like tuning for performance based on usage. Using short-lived processes in Erlang is using Erlang for what it is good at.... why would you fight that by having code blow-up memory-wise by leaving long-lived Erlang processes in your source code?
This usage of short-lived processes is what concerns me when we talk about how Erlang pids are limited to 2^28 unique pids per node (i.e., 28 bits, not yet changed to be 60 bits), but it is good that the new algorithm for pid reuse in R16B01 will support this approach (i.e., the Erlang pid per rpc-call approach). (details are from the "pid representation in external term format" email thread)
On 09/17/2013 08:03 AM, Lukas Larsson wrote:
> as far as I can see erlang:garbage_collect/0,1 does do fullsweeps [1].
>
> [1]: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/bif.c#L3770
>
>
> On Tue, Sep 17, 2013 at 4:53 PM, Robert Virding <robert.virding@REDACTED <mailto:robert.virding@REDACTED>> wrote:
>
> OK, I meant doing an explicit garbage collectionen and was under the impression that calling erlang:garbage_collect() actually did a full sweep of the process which is why hibernating seemed a bit of an overkill.
>
> Rober
>
> ----- Original Message -----
> > From: "Fred Hebert" <mononcqc@REDACTED <mailto:mononcqc@REDACTED>>
> > To: "Robert Virding" <robert.virding@REDACTED <mailto:robert.virding@REDACTED>>
> > Cc: "Erlang Questions Mailing List" <erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>>
> > Sent: Tuesday, 17 September, 2013 4:36:19 PM
> > Subject: Re: [erlang-questions] Problem with Beam process not freeing memory
> >
> > Hi Robert,
> >
> > I CC'd the mailing list on this post, because I felt it could be
> > interesting to share with everyone.
> >
> > To get into some more details, the difference between a
> > garbage-collection and hibernating is that hibernation forces a
> > full-sweep and does compaction work. It is more likely to actually
> > remove old refs to binaries.
> >
> > The tricky part about binary leaks in these cases is that if the process
> > you're garbage collecting holds some very long-lived references (or
> > takes a long while before enabling them), you will move the references
> > to the old heap, if my understanding is correct. Now if your process
> > that leaks resources is busy or bogged down by some task, there are
> > chances that manually calling GC at higher frequencies will force
> > short-lived references to the old heap. Eventually, most of the
> > subsequent GCs are done for no good reason until there's a full sweep to
> > free the references, if my understanding is right.
> >
> > In comparison, some hibernations may turn out to be beneficial due to
> > how they do the full sweep, especially on less active processes, without
> > changing spawn_opt values or whatever.
> >
> > Both cases are not necessarily great, and I don't think there's one easy
> > way to deal with it.
> >
> > One thing I've had in mind to try for a while was to run a function a
> > bit like:
> >
> > -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer().
> > gc_count(PreviousCounter, Bin) ->
> > case byte_size(Bin) of
> > N when N >= 64 -> % refc binary
> > Count = N + PreviousCounter
> > case NewCount >= ?THRESHOLD of
> > true ->
> > erlang:garbage_collect(),
> > 0;
> > false ->
> > NewCount
> > end;
> > N -> % heap binary
> > PreviousCounter+N
> > end.
> >
> > that could alternatively force some hibernation instead of GC'ing. This
> > one could basically track the size of all binaries seen manually and
> > force something when you go over a certain amount. It sucks, though,
> > because that's basically manually doing your collection, and it doesn't
> > mean that because you have seen a binary, it's ready to be GC'd. I've
> > thus avoided trying it in the real world for now.
> >
> > In practice, at Heroku, we've decided to go for a hybrid approach in
> > logplex. We force hibernation on some important events that interrupt
> > our work flow no matter what (such as a socket disconnection, or long
> > periods of time [seconds] without activity for a process), and have put
> > a workaround in place to force VM-wide GCs when we're reaching critical
> > amounts of memory:
> > https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl
> >
> > The objective was to use global GC as a last measure in case individual
> > (unobtrusive) hibernates were not enough to save a node.
> >
> > This later on prompted for exploring the allocators of the VM -- the
> > value used (erlang:memory(total)) didn't represent the OS-imposed limits
> > on the VM: nodes would be killed by going out of memory without first
> > having had the chance to run the global GC. This lead to discovering
> > things about fragmentation and characterizing our workloads to pick
> > better allocation strategies that seem to work decently so far, so that
> > erlang:memory(total), for one, has the right values, and also that we
> > have a better time releasing allocated blocks of memory when most
> > binaries vanish.
> >
> > I hope we can remove both the artificial hibernation calls and the
> > workarounds to force some global GCs in the near future. Ideally, it
> > sounds like the VM should possibly do more when it comes to the weight
> > of refc binaries to individual processes' memory for GC, but I don't
> > have a good idea of how this should be done in practice without having
> > elephant-sized assumptions and holes in the solution without adding more
> > knobs to the VM to configure things. Plus I'd have no idea on how to
> > actually implement it.
> >
> > Regards,
> > Fred.
> >
> >
> > On 09/17, Robert Virding wrote:
> > > Hi Fred,
> > >
> > > You recommend hibernating a process. Do you think this is better than
> > > calling the garbage collector in a process? I have no idea but hibernating
> > > seems more drastic, especially if the process is "in use"?
> > >
> > > Robert
> > >
> > > ----- Original Message -----
> > > > From: "Fred Hebert" <mononcqc@REDACTED <mailto:mononcqc@REDACTED>>
> > > > To: "Tino Breddin" <tino.breddin@REDACTED <mailto:tino.breddin@REDACTED>>
> > > > Cc: "Erlang Questions Mailing List" <erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>>
> > > > Sent: Tuesday, 17 September, 2013 2:39:41 AM
> > > > Subject: Re: [erlang-questions] Problem with Beam process not freeing
> > > > memory
> > > >
> > > > I've recently run in similar issues and have received a bit of help from
> > > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me,
> > > > I tried to put into the recon library, currently on a different branch
> > > > awaiting review and prone to change:
> > > > https://github.com/ferd/recon/tree/allocators
> > > >
> > > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)`
> > > > where `N` is the number of 'highest results' you want. The function will
> > > > take a snapshot of the number of binary refs in your processes, then GC
> > > > the node entirely, and then take another snapshot, make diff, and return
> > > > the N biggest deltas in binaries. This will let you know what processes
> > > > hold the most references to stale refc binaries. I recommend hibernation
> > > > as a first way to try and fix this if it is the problem.
> > > >
> > > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)`
> > > > where `Arg` can be:
> > > > - `used` for the memory actively used (i.e. erlang:memory(total))
> > > > - `allocated` for the memory reserved by individual allocators (sum)
> > > > - `usage` for the percentage.
> > > > If the result of `allocated` is close to what the OS reports, you
> > > > probably have fragmentation issues. If not, you may have a NIF or driver
> > > > that allocates data outside of the ERTS allocators
> > > >
> > > > 3. check individual allocator usage levels with
> > > > `recon_alloc:fragmentation(current)`. It will return usage percentages
> > > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes
> > > > by default. When the data allocated is too large (> than the single
> > > > block carrier threshold [sbct]), it goes into its own block. Compare the
> > > > results with what you get with `recon_alloc:fragmentation(max)`. If the
> > > > current values have very low usage but the max ones have large ones, you
> > > > may have lingering data, possibly held in long-term references or
> > > > whatever that blocks deallocation of specific carriers. Different
> > > > carrier strategies can help, which we can dive into if you see a problem
> > > > with this.
> > > >
> > > > Feel free to read the comments in `recon_alloc` until I actually merge
> > > > it in master, they contain some of the details about what to do or look
> > > > for.
> > > >
> > > > Lukas may want me to correct me on the content of this post. I'm going
> > > > from the limited knowledge he transmitted to me here, or rather, my
> > > > limited understanding of it :)
> > > >
> > > > Regards,
> > > > Fred.
> > > >
> > > > On 09/16, Tino Breddin wrote:
> > > > > Hi list,
> > > > >
> > > > > I'm experiencing issues with a couple of Beam nodes where I see a huge
> > > > > gap
> > > > > between the node's reported memory usage and the underlying Linux
> > > > > Kernel's
> > > > > view.
> > > > >
> > > > > This is using R15B01.
> > > > >
> > > > > As a start an application in such a node stores a lot of tuples
> > > > > (containing
> > > > > atoms and binary data) in ETS tables. That proceeds until a point where
> > > > > memory usage is 70% (6GB) of the available memory. At that point
> > > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the
> > > > > memory
> > > > > usage. Then an internal cleanup task is performed, which clears
> > > > > obsolete
> > > > > records from the ETS tables. Afterwards, erlang:memory() reports an
> > > > > expected low value of roughly 60MB memory usage. (This includes binary
> > > > > data). However, the kernel still reports the high memory usage values
> > > > > (both
> > > > > VmRss and VmTotal) for the node. The kernel's usage view will stay
> > > > > stable
> > > > > until the ETS tables are filled to a point where the real memory usage
> > > > > exceeds the kernel's view, then the kernel reported usage will grow as
> > > > > well.
> > > > >
> > > > > Now having checked the node in some details I'm wondering what causes
> > > > > this
> > > > > difference between the BEAM's view and the Kernel's view on memory
> > > > > usage. I
> > > > > have 2 ideas which I'm checking right now.
> > > > >
> > > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed
> > > > > because
> > > > > the original dispatcher process which it was passed through before
> > > > > being
> > > > > stored in an ETS table is still alive. Thus there is still some
> > > > > reference
> > > > > to it? However, this would not explain why erlang:memory() reports a
> > > > > very
> > > > > low value for used memory for binaries.
> > > > >
> > > > > (2) low-level memory leak: Some driver or NIF leaking memory, which
> > > > > would
> > > > > obviously not be reported by erlang:memory(). However, then it
> > > > > surprises me
> > > > > that the Kernel's view stays stable while the BEAM's actual memory
> > > > > usage is
> > > > > still below the Kernel's view. It should be continuously growing in
> > > > > this
> > > > > case imho.
> > > > >
> > > > > I'd appreciate if anyone has some more insight or experience with such
> > > > > a
> > > > > behaviour, while I'm further digging into this.
> > > > >
> > > > > Cheers,
> > > > > Tino
> > > >
> > > > > _______________________________________________
> > > > > erlang-questions mailing list
> > > > > erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> > > > > http://erlang.org/mailman/listinfo/erlang-questions
> > > >
> > > > _______________________________________________
> > > > erlang-questions mailing list
> > > > erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> > > > http://erlang.org/mailman/listinfo/erlang-questions
> > > >
> >
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130917/259b623a/attachment.htm>
More information about the erlang-questions
mailing list