Unified heap

[Francesco Cesarini]

Sorry about that. I should have expressed myself a little better and gone
into more detail. I came into contact with Erlang memory fragmentation in
1998. A project started stress testing the system with huge configurations.
This system had ets tables that stored the board identifiers as strings. As
thousands of boards where configured, thousands of copies of these
identifiers where made. New entries where added, deleted, reconfigured.
This became, in due time, a serious cause for fragmentation due to the size
of the strings involved. (The problem was later solved with the help of the
OTP guys, by creating a separate table with the board types indexing them
with integers). If I am
not mistaken, at the time, ets tables where stored on a separate heap (We
might be talking R3 - R5 here), probably even then, scattered all over the
memory.

From what I have understood on the discussions & results on the list so
far, in HIPE, ets tables are now stored on the unified heap through a new
scheme, and I assume they are clustered. (Sounds a little strange, but that
is the only explanation I have to Pascal’s results, unless enabling the
unified heap also does something with the ets tables scheme on a  separate
heap). Isn’t clustering together lots of small memory chucks a reason
enough for worry, especially if they are located alongside process data?

I am just speculating, as I have not done any tests yet. But considering
that we almost scratched out heads bald trying to figure out the problem
back in 1998 (It took some time as it only occurred on large systems which
had been running a few days), the worry is always there. That is why I was
wondering if the hipe group has given any thought to these problems.

Regards,
Francesco

Kostis Sagonas wrote:

> Francesco Cesarini wrote:
>
>  > When looking at the unified heap, one of my major worries is memory
>  > fragmentation. Due to past experience (Jam 1998) after having run
>  > heavy duty programs for a long period of time, I am scared it is
>  > bound to happen.
>
> You should explain to us a bit more why you are worried about memory
> fragmentation in the unified heap model, especially since (at least
> in theory) one of the arguments "pro" unified heap is that of lower
> fragmentation.  The reason is that with the unified heap model, the
> whole memory is available to any process that needs it: either
> immediately if there is free space, or after GC.  In the current
> memory model, a process cannot allocate in the heap of another
> process even if that heap contains a large area of unused space.
>
>  > Has the Hipe team looked into this?  Have you any theories or done
>  > any measurements?
>
> For theories, see above.  For measurements, we are currently conducting
> them, so if you have a program that supports your worries, please do
> send it to us.
>
> Cheers,
>
> Kostis.

--
http://www.erlang-consulting.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: francesco.vcf
Type: text/x-vcard
Size: 352 bytes
Desc: Card for Francesco Cesarini
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20011011/f32a0513/attachment.vcf>