[erlang-questions] Erlang and Memory management

[Gleb Peregud]

Erlang has generational per process GC. This means that after many read
operations where you are likely to create new small terms (e.g. reply tuple
with value), the GC may kick in. Generational GC would copy all reachable
terms from old heap to new one, before discarding old heap. This leads to
double use of memory, which might be the reason of crashes.

Best,
Gleb
On 31 May 2011 21:29, "Paul Meier" <pablo.a.meier@REDACTED> wrote:
> I'm in a similar situation. I've got a few large tries, implemented
> with gb_trees. I keep them in an orddict, which I'm using as the
> State variable of a gb_server. For example:
>
> ------
> init() ->
> %% State is an orddict of 3 giant Tries, pre-made and fetched from
> the filesystem.
> {ok, State}.
>
> handle_call({keyed_operation_on_large_data, Key}, _From, State) ->
> GiantTrie = orddict:find(Key, State),
> .... computations using GiantTrie....
> {reply, Result, State}.
>
> I'm finding that my process is crashing due to inability to mmap, and
> the erl_crash.dump file tells me that the gen_server's supervisor (or,
> on occasion, the error_logger) is the one with a prohibitively large
> Stack+Heap. Also, I never modify the Tries or the orddict containing
> them in calls to the gb_server: it's all read-only.
>
> Is storing the value in an orddict, or using regular terms as opposed
> to binaries causing unnecessary copying? While _I_ know the data is
> being used in a read-only fashion, is there a place where copying
> might be occurring because the VM doesn't know this?
>
> I was hoping to play with binaries as a next attempt at fixing this
> (the links in this thread suggest that large binaries are handled more
> conservatively), but this thread appeared fortuitously, and maybe
> someone can spot the very likely obvious error that I'm making.
>
> I'm grateful for your help, whether in this thread or the others I've
> enjoyed reading ^_^
>
> -Paul
>
>
> On Tue, May 31, 2011 at 11:19 AM, Kostis Sagonas <kostis@REDACTED>
wrote:
>> Robert Virding wrote:
>>>
>>> I just want to point out that these rules are BEAM specific and not
Erlang
>>> specific.
>>
>> Well, this is not correct: the BEAM nowhere specifies that its
instruction
>> set has to be implemented using a private heap architecture.  In fact, we
>> successfully used the BEAM instruction set to implement both a shared
heap
>> and a hybrid heap implementation.  See:
>>
>>  http://user.it.uu.se/~kostis/Papers/heap.pdf
>>
>>> So on an implementation with a single heap all data is shared. Now there
>>> aren't currently many single heap implementations, to be exact only one,
>>> erjang (Erlang on the JVM). :-)
>>
>> There have been more.  We had an OTP system with a shared heap and the
>> hybrid heap system was part of OTP for quite a while.  IMO, it's too bad
>> that it was not maintained when Erlang/OTP was extended to support SMP
>> architectures.
>>
>> Also, the ETOS (Erlang to Scheme) system was based on a shared heap
>> architecture.
>>
>> Kostis
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110601/98eaab02/attachment.htm>