[erlang-questions] hybrid heap question

[Kenneth Lundin]

Hi,

Yes I agree with Kostis that the GC is not incremental  in the
separate heap solution either and it is possible to get a very long GC
there as well.
In practice there seems to be very little problems with this.
The situation is actually better in the  SMP emulator running with a
number of scheduler threads because only the scheduler thread running
the process whose heap is garbage collected will be occupied with the
possibly big GC. All other scheduler threads can execute Erlang
processes and even send messages to the process which is under GC.

Because the hybrid heap involves a heap which is shared between all processes
it is harder to do a GC without impact on all processes during that time.
Incremental GC is therefore more of a necessity for Hybrid heap than for
the separate heap solution at least in the SMP emulator.
Implementing the incremental GC for the SMP emulator will also be harder than
for the non SMP emulator I think.

I might whoever be wrong in the assumption  that it is more likely to
get a very big collection of data in one GC with the Hybrid heap.

/Kenneth

On 2/28/07, Kostis Sagonas <kostis@REDACTED> wrote:
> Kenneth Lundin wrote:
> > Hi,
> >
> >>From our view (OTP team at Ericsson) the Hybrid heap version of the
> > emulator is experimental. With expreimental I mean that:
> >  - it is definately
> > not as stable and well tested as the mature separate heap solution.
> > - it is not used in any products as far as I know.
> > - The hybrid heap can be a very good solution when it is finished. The lack of
> > compiler support (identifying data that will be sent in a message and
> > can be put on shared heap from the beginning) and the lack of
> > incremental GC makes it less useful in real products. There will for
> > example be very long GC's in the current system, which interrupts all
> > other execution in the VM.
> > - We have no immediate plans to make Hybrid heap and SMP work together
> > for several reasons. 1) The separate heap and SMP works very very well
> > and  we see no problems with using that for any type of application.
> > 2) We are dependent on the competence from the HiPE group in Uppsala
> > University regarding the Hybrid heap ,we will happily include their
> > contribution if
> > they make Hybrid heap work together with SMP , add incremental GC etc.
> >
> > In summary: Today the separate heap (default) solution is the stable
> > and recommended one for both SMP and non SMP cases.
> > Hybrid heap is experimental from our point of view. Might be very
> > stable on non SMP but that is nothing we guarantee.
> >
> > /Regards Kenneth (OTP team at Ericsson)
>
> As somebody very involved in the design and implementation of the Hybrid
> heap version of the Erlang emulator, I just want to point out for the
> record that the following:
>
>   > There will for example be very long GC's in the current system,
>   > which interrupts all other execution in the VM.
>
> is a general "feature" in Erlang/OTP and is not specific to the Hybrid
> vs. Private heap implementation.  Currently, the garbage collector of
> the Erlang/OTP system is not incremental and cannot be preempted.  This
> means that there is absolutely no guarantee that GC will not take
> arbitrary long time.  Again for the record, I have to point out that
> this does *not* seem to be a big problem in practice -- GC in Erlang/OTP
> is actually very fast most of the times.  However, that there is no
> bounded GC time guarantee holds independently of the architecture of the
> system and I know of no fundamental reasons why the situation is in
> principle worse in the Hybrid heap system.  In fact, it can actually be
> better because the Hybrid heap system fundamentally shares data which is
> otherwise copied (possibly multiple times) when processes communicate in
> a Private heap environment.  Again, this worst case situation does not
> seem to occur very often in practice either.
>
> Kostis (HiPE team)
>