[erlang-questions] Erlang and Memory management

[Paul Meier]

I'm in a similar situation.  I've got a few large tries, implemented
with gb_trees.  I keep them in an orddict, which I'm using as the
State variable of a gb_server.  For example:

------
init() ->
    %% State is an orddict of 3 giant Tries, pre-made and fetched from
the filesystem.
    {ok, State}.

handle_call({keyed_operation_on_large_data, Key}, _From, State) ->
    GiantTrie = orddict:find(Key, State),
    .... computations using GiantTrie....
    {reply, Result, State}.

I'm finding that my process is crashing due to inability to mmap, and
the erl_crash.dump file tells me that the gen_server's supervisor (or,
on occasion, the error_logger) is the one with a prohibitively large
Stack+Heap.  Also, I never modify the Tries or the orddict containing
them in calls to the gb_server: it's all read-only.

Is storing the value in an orddict, or using regular terms as opposed
to binaries causing unnecessary copying?  While _I_ know the data is
being used in a read-only fashion, is there a place where copying
might be occurring because the VM doesn't know this?

I was hoping to play with binaries as a next attempt at fixing this
(the links in this thread suggest that large binaries are handled more
conservatively), but this thread appeared fortuitously, and maybe
someone can spot the very likely obvious error that I'm making.

I'm grateful for your help, whether in this thread or the others I've
enjoyed reading ^_^

-Paul


On Tue, May 31, 2011 at 11:19 AM, Kostis Sagonas <kostis@REDACTED> wrote:
> Robert Virding wrote:
>>
>> I just want to point out that these rules are BEAM specific and not Erlang
>> specific.
>
> Well, this is not correct: the BEAM nowhere specifies that its instruction
> set has to be implemented using a private heap architecture.  In fact, we
> successfully used the BEAM instruction set to implement both a shared heap
> and a hybrid heap implementation.  See:
>
>  http://user.it.uu.se/~kostis/Papers/heap.pdf
>
>> So on an implementation with a single heap all data is shared. Now there
>> aren't currently many single heap implementations, to be exact only one,
>> erjang (Erlang on the JVM). :-)
>
> There have been more.  We had an OTP system with a shared heap and the
> hybrid heap system was part of OTP for quite a while.  IMO, it's too bad
> that it was not maintained when Erlang/OTP was extended to support SMP
> architectures.
>
> Also, the ETOS (Erlang to Scheme) system was based on a shared heap
> architecture.
>
> Kostis
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>