[erlang-questions] Erlang and Memory management

[Michael Truog]

A better trie data structure exists that should consume less memory, here:
https://github.com/okeuday/trie

On 05/31/2011 12:28 PM, Paul Meier wrote:
> I'm in a similar situation.  I've got a few large tries, implemented
> with gb_trees.  I keep them in an orddict, which I'm using as the
> State variable of a gb_server.  For example:
>
> ------
> init() ->
>     %% State is an orddict of 3 giant Tries, pre-made and fetched from
> the filesystem.
>     {ok, State}.
>
> handle_call({keyed_operation_on_large_data, Key}, _From, State) ->
>     GiantTrie = orddict:find(Key, State),
>     .... computations using GiantTrie....
>     {reply, Result, State}.
>
> I'm finding that my process is crashing due to inability to mmap, and
> the erl_crash.dump file tells me that the gen_server's supervisor (or,
> on occasion, the error_logger) is the one with a prohibitively large
> Stack+Heap.  Also, I never modify the Tries or the orddict containing
> them in calls to the gb_server: it's all read-only.
>
> Is storing the value in an orddict, or using regular terms as opposed
> to binaries causing unnecessary copying?  While _I_ know the data is
> being used in a read-only fashion, is there a place where copying
> might be occurring because the VM doesn't know this?
>
> I was hoping to play with binaries as a next attempt at fixing this
> (the links in this thread suggest that large binaries are handled more
> conservatively), but this thread appeared fortuitously, and maybe
> someone can spot the very likely obvious error that I'm making.
>
> I'm grateful for your help, whether in this thread or the others I've
> enjoyed reading ^_^
>
> -Paul
>
>
> On Tue, May 31, 2011 at 11:19 AM, Kostis Sagonas <kostis@REDACTED> wrote:
>> Robert Virding wrote:
>>> I just want to point out that these rules are BEAM specific and not Erlang
>>> specific.
>> Well, this is not correct: the BEAM nowhere specifies that its instruction
>> set has to be implemented using a private heap architecture.  In fact, we
>> successfully used the BEAM instruction set to implement both a shared heap
>> and a hybrid heap implementation.  See:
>>
>>  http://user.it.uu.se/~kostis/Papers/heap.pdf
>>
>>> So on an implementation with a single heap all data is shared. Now there
>>> aren't currently many single heap implementations, to be exact only one,
>>> erjang (Erlang on the JVM). :-)
>> There have been more.  We had an OTP system with a shared heap and the
>> hybrid heap system was part of OTP for quite a while.  IMO, it's too bad
>> that it was not maintained when Erlang/OTP was extended to support SMP
>> architectures.
>>
>> Also, the ETOS (Erlang to Scheme) system was based on a shared heap
>> architecture.
>>
>> Kostis
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>