[erlang-questions] Garbage collection

[Lukas Larsson]

On Fri, Oct 31, 2014 at 11:06 AM, Chandru <
chandrashekhar.mullaparthi@REDACTED> wrote:

>
> On 31 October 2014 10:01, Lukas Larsson <lukas@REDACTED> wrote:
>
>> On Fri, Oct 31, 2014 at 10:51 AM, Chandru <
>> chandrashekhar.mullaparthi@REDACTED> wrote:
>>
>>> Thank you Lukas.
>>>
>>> On 31 October 2014 09:37, Lukas Larsson <lukas@REDACTED> wrote:
>>>
>>>> Hello,
>>>>
>>>> On Fri, Oct 31, 2014 at 10:20 AM, Chandrashekhar Mullaparthi <
>>>> chandrashekhar.mullaparthi@REDACTED> wrote:
>>>>
>>>>>
>>>>>
>>>>> I have a question about beam’s GC implementation. When an erlang
>>>>> process is being GCed, is the processing required to do the GC taken out of
>>>>> the process’s 2000 reduction quota, or is it done after a process has been
>>>>> scheduled out?
>>>>>
>>>>>
>>>>>
>>>>
>>>> The GC work is taken out of the process' reductions. The GC is never
>>>> triggered when it is scheduled out, but it can be triggered before being
>>>> scheduled in, in which case the newly allotted reductions will be reduced
>>>> by the GC work.
>>>>
>>>
>>> So what happens if the process has a large heap? Can the GC end up
>>> taking more time than to execute 2000 reductions? Or is it somehow time
>>> bounded? If it is not time bounded, it explains a lot of the problems I'm
>>> seeing on a system.
>>>
>>
>> The current GC is not incremental, so once it has started doing work it
>> cannot be interrupted. This means that if a process has a large heap it
>> will block all other execution on that scheduler for the GC duration.
>>
>
> Can it also block other schedulers by any chance? Robert Virding's
> presentation [1] says that every 20-40k reductions, a new master scheduler
> is chosen. I'm wondering if this transition of master scheduler has to
> happen while one of the scheduler's is stuck in a long GC, will it
> potentially block other schedulers?
>
>
It can result in schedulers not waking up properly, for the same reasons as
long running nifs/bifs causes this. So if this is you problem I would have
a look at which processes are using a lot of heap space and try to reduce
it, or make sure that they do not GC :)

Scott has collected a bunch of his observations on the long running
nifs/bifs issue here:
https://github.com/slfritchie/nifwait/blob/md5/README.md. Sometime in the
not too distant future I hope to have the time to write an incremental GC
for large Erlang heaps, but in the meantime I believe Scott recommends
using something like "+sfwi 500 +scl false" in order to avoid this problem.
Try them out and see if the options work for you.

Lukas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141031/3431f4dc/attachment.htm>