[erlang-questions] Problem with Beam process not freeing memory

J K jmakarlsson@REDACTED
Tue Sep 17 17:46:24 CEST 2013 

Hi,
I think you need to set the fullsweep_after option to 0, for example:  spawn_opt(Node, M, F, [Args,self(),File],[{fullsweep_after,0}]) 
for it to happen immediately. Otherwise it will happen some time later.

Regarding:
 nodes would be killed by going out of memory without first
> having had the chance to run the global GC.


I also ran into this problem and had to manually insert erlang:garbage_collect() at strategic places in my code for my application to work. I would prefer an option to say that "aggressive" GC should be done before terminating the node.

JOhan



>________________________________
> From: Lukas Larsson <garazdawi@REDACTED>
>To: Robert Virding <robert.virding@REDACTED> 
>Cc: Erlang Questions Mailing List <erlang-questions@REDACTED> 
>Sent: Tuesday, September 17, 2013 5:03 PM
>Subject: Re: [erlang-questions] Problem with Beam process not freeing memory
> 
>
>
>as far as I can see erlang:garbage_collect/0,1 does do fullsweeps [1].
>
>   [1]: https://github.com/erlang/otp/blob/maint/erts/emulator/beam/bif.c#L3770
>
>
>
>
>On Tue, Sep 17, 2013 at 4:53 PM, Robert Virding <robert.virding@REDACTED> wrote:
>
>OK, I meant doing an explicit garbage collectionen and was under the impression that calling erlang:garbage_collect() actually did a full sweep of the process which is why hibernating seemed a bit of an overkill.
>>
>>Rober
>>
>>
>>----- Original Message -----
>>> From: "Fred Hebert" <mononcqc@REDACTED>
>>
>>> To: "Robert Virding" <robert.virding@REDACTED>
>>> Cc: "Erlang Questions Mailing List" <erlang-questions@REDACTED>
>>
>>> Sent: Tuesday, 17 September, 2013 4:36:19 PM
>>> Subject: Re: [erlang-questions] Problem with Beam process not freeing memory
>>>
>>> Hi Robert,
>>>
>>> I CC'd the mailing list on this post, because I felt it could be
>>> interesting to share with everyone.
>>>
>>> To get into some more details, the difference between a
>>> garbage-collection and hibernating is that hibernation forces a
>>> full-sweep and does compaction work. It is more likely to actually
>>> remove old refs to binaries.
>>>
>>> The tricky part about binary leaks in these cases is that if the process
>>> you're garbage collecting holds some very long-lived references (or
>>> takes a long while before enabling them), you will move the references
>>> to the old heap, if my understanding is correct. Now if your process
>>> that leaks resources is busy or bogged down by some task, there are
>>> chances that manually calling GC at higher frequencies will force
>>> short-lived references to the old heap.  Eventually, most of the
>>> subsequent GCs are done for no good reason until there's a full sweep to
>>> free the references, if my understanding is right.
>>>
>>> In comparison, some hibernations may turn out to be beneficial due to
>>> how they do the full sweep, especially on less active processes, without
>>> changing spawn_opt values or whatever.
>>>
>>> Both cases are not necessarily great, and I don't think there's one easy
>>> way to deal with it.
>>>
>>> One thing I've had in mind to try for a while was to run a function a
>>> bit like:
>>>
>>> -spec gc_count(non_neg_integer(), binary()) -> non_neg_integer().
>>> gc_count(PreviousCounter, Bin) ->
>>>     case byte_size(Bin) of
>>>         N when N >= 64 -> % refc binary
>>>             Count = N + PreviousCounter
>>>             case NewCount >= ?THRESHOLD of
>>>                 true ->
>>>                     erlang:garbage_collect(),
>>>                     0;
>>>                false ->
>>>                    NewCount
>>>             end;
>>>         N -> % heap binary
>>>             PreviousCounter+N
>>>     end.
>>>
>>> that could alternatively force some hibernation instead of GC'ing. This
>>> one could basically track the size of all binaries seen manually and
>>> force something when you go over a certain amount. It sucks, though,
>>> because that's basically manually doing your collection, and it doesn't
>>> mean that because you have seen a binary, it's ready to be GC'd. I've
>>> thus avoided trying it in the real world for now.
>>>
>>> In practice, at Heroku, we've decided to go for a hybrid approach in
>>> logplex. We force hibernation on some important events that interrupt
>>> our work flow no matter what (such as a socket disconnection, or long
>>> periods of time [seconds] without activity for a process), and have put
>>> a workaround in place to force VM-wide GCs when we're reaching critical
>>> amounts of memory:
>>> https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl
>>>
>>> The objective was to use global GC as a last measure in case individual
>>> (unobtrusive) hibernates were not enough to save a node.
>>>
>>> This later on prompted for exploring the allocators of the VM -- the
>>> value used (erlang:memory(total)) didn't represent the OS-imposed limits
>>> on the VM: nodes would be killed by going out of memory without first
>>> having had the chance to run the global GC. This lead to discovering
>>> things about fragmentation and characterizing our workloads to pick
>>> better allocation strategies that seem to work decently so far, so that
>>> erlang:memory(total), for one, has the right values, and also that we
>>> have a better time releasing allocated blocks of memory when most
>>> binaries vanish.
>>>
>>> I hope we can remove both the artificial hibernation calls and the
>>> workarounds to force some global GCs in the near future. Ideally, it
>>> sounds like the VM should possibly do more when it comes to the weight
>>> of refc binaries to individual processes' memory for GC, but I don't
>>> have a good idea of how this should be done in practice without having
>>> elephant-sized assumptions and holes in the solution without adding more
>>> knobs to the VM to configure things. Plus I'd have no idea on how to
>>> actually implement it.
>>>
>>> Regards,
>>> Fred.
>>>
>>>
>>> On 09/17, Robert Virding wrote:
>>> > Hi Fred,
>>> >
>>> > You recommend hibernating a process. Do you think this is better than
>>> > calling the garbage collector in a process? I have no idea but hibernating
>>> > seems more drastic, especially if the process is "in use"?
>>> >
>>> > Robert
>>> >
>>> > ----- Original Message -----
>>> > > From: "Fred Hebert" <mononcqc@REDACTED>
>>> > > To: "Tino Breddin" <tino.breddin@REDACTED>
>>> > > Cc: "Erlang Questions Mailing List" <erlang-questions@REDACTED>
>>> > > Sent: Tuesday, 17 September, 2013 2:39:41 AM
>>> > > Subject: Re: [erlang-questions] Problem with Beam process not freeing
>>> > > memory
>>> > >
>>> > > I've recently run in similar issues and have received a bit of help from
>>> > > Lukas Larsson, which I'm glad to pass on to you. Whatever he taught me,
>>> > > I tried to put into the recon library, currently on a different branch
>>> > > awaiting review and prone to change:
>>> > > https://github.com/ferd/recon/tree/allocators
>>> > >
>>> > > 1. Checking for binary leaks, I recommend calling `recon:bin_leak(N)`
>>> > > where `N` is the number of 'highest results' you want. The function will
>>> > > take a snapshot of the number of binary refs in your processes, then GC
>>> > > the node entirely, and then take another snapshot, make diff, and return
>>> > > the N biggest deltas in binaries. This will let you know what processes
>>> > > hold the most references to stale refc binaries. I recommend hibernation
>>> > > as a first way to try and fix this if it is the problem.
>>> > >
>>> > > 2. Check the reported/allocated memory with `recon_alloc:memory(Arg)`
>>> > > where `Arg` can be:
>>> > >  - `used` for the memory actively used (i.e. erlang:memory(total))
>>> > >  - `allocated` for the memory reserved by individual allocators (sum)
>>> > >  - `usage` for the percentage.
>>> > > If the result of `allocated` is close to what the OS reports, you
>>> > > probably have fragmentation issues. If not, you may have a NIF or driver
>>> > > that allocates data outside of the ERTS allocators
>>> > >
>>> > > 3. check individual allocator usage levels with
>>> > > `recon_alloc:fragmentation(current)`. It will return usage percentages
>>> > > for mbcs and sbcs. Mbcs are multiblock carriers and are where data goes
>>> > > by default. When the data allocated is too large (> than the single
>>> > > block carrier threshold [sbct]), it goes into its own block. Compare the
>>> > > results with what you get with `recon_alloc:fragmentation(max)`. If the
>>> > > current values have very low usage but the max ones have large ones, you
>>> > > may have lingering data, possibly held in long-term references or
>>> > > whatever that blocks deallocation of specific carriers. Different
>>> > > carrier strategies can help, which we can dive into if you see a problem
>>> > > with this.
>>> > >
>>> > > Feel free to read the comments in `recon_alloc` until I actually merge
>>> > > it in master, they contain some of the details about what to do or look
>>> > > for.
>>> > >
>>> > > Lukas may want me to correct me on the content of this post. I'm going
>>> > > from the limited knowledge he transmitted to me here, or rather, my
>>> > > limited understanding of it :)
>>> > >
>>> > > Regards,
>>> > > Fred.
>>> > >
>>> > > On 09/16, Tino Breddin wrote:
>>> > > > Hi list,
>>> > > >
>>> > > > I'm experiencing issues with a couple of Beam nodes where I see a huge
>>> > > > gap
>>> > > > between the node's reported memory usage and the underlying Linux
>>> > > > Kernel's
>>> > > > view.
>>> > > >
>>> > > > This is using R15B01.
>>> > > >
>>> > > > As a start an application in such a node stores a lot of tuples
>>> > > > (containing
>>> > > > atoms and binary data) in ETS tables. That proceeds until a point where
>>> > > > memory usage is 70% (6GB) of the available memory. At that point
>>> > > > erlang:memory() and top (or /proc/PID/status) agree roughly on the
>>> > > > memory
>>> > > > usage. Then an internal cleanup task is performed, which clears
>>> > > > obsolete
>>> > > > records from the ETS tables. Afterwards, erlang:memory() reports an
>>> > > > expected low value of roughly 60MB memory usage. (This includes binary
>>> > > > data). However, the kernel still reports the high memory usage values
>>> > > > (both
>>> > > > VmRss and VmTotal) for the node. The kernel's usage view will stay
>>> > > > stable
>>> > > > until the ETS tables are filled to a point where the real memory usage
>>> > > > exceeds the kernel's view, then the kernel reported usage will grow as
>>> > > > well.
>>> > > >
>>> > > > Now having checked the node in some details I'm wondering what causes
>>> > > > this
>>> > > > difference between the BEAM's view and the Kernel's view on memory
>>> > > > usage. I
>>> > > > have 2 ideas which I'm checking right now.
>>> > > >
>>> > > > (1) Not GC'ed binaries: Could it be that binary data is not GC'ed
>>> > > > because
>>> > > > the original dispatcher process which it was passed through before
>>> > > > being
>>> > > > stored in an ETS table is still alive. Thus there is still some
>>> > > > reference
>>> > > > to it? However, this would not explain why erlang:memory() reports a
>>> > > > very
>>> > > > low value for used memory for binaries.
>>> > > >
>>> > > > (2) low-level memory leak: Some driver or NIF leaking memory, which
>>> > > > would
>>> > > > obviously not be reported by erlang:memory(). However, then it
>>> > > > surprises me
>>> > > > that the Kernel's view stays stable while the BEAM's actual memory
>>> > > > usage is
>>> > > > still below the Kernel's view. It should be continuously growing in
>>> > > > this
>>> > > > case imho.
>>> > > >
>>> > > > I'd appreciate if anyone has some more insight or experience with such
>>> > > > a
>>> > > > behaviour, while I'm further digging into this.
>>> > > >
>>> > > > Cheers,
>>> > > > Tino
>>> > >
>>> > > > _______________________________________________
>>> > > > erlang-questions mailing list
>>> > > > erlang-questions@REDACTED
>>> > > > http://erlang.org/mailman/listinfo/erlang-questions
>>> > >
>>> > > _______________________________________________
>>> > > erlang-questions mailing list
>>> > > erlang-questions@REDACTED
>>> > > http://erlang.org/mailman/listinfo/erlang-questions
>>> > >
>>>
>>_______________________________________________
>>erlang-questions mailing list
>>erlang-questions@REDACTED
>>http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>_______________________________________________
>erlang-questions mailing list
>erlang-questions@REDACTED
>http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130917/bc0bde58/attachment.htm>

More information about the erlang-questions mailing list