[erlang-questions] dealing with large dicts

[Richard Carlsson]

Paul Fisher wrote:
> Did you notice that i suggested that he use the 'compressed' option in
> the term_to_binary/2 call?  While this does not avoid all of the
> overhead it does have much less data moving between the processes, which
> will help some.

I missed the compressed flag, but it will not do much for performance,
even if it might postpone the memory problem. (A quick test compressing
a large dict reduced the binary to a quarter of the uncompressed size.)

I looked back to the original post, which said the problem was an out
of memory error - and assuming that a single dict (or two) fit in
memory, the code example cannot in itself be the problem. It is very
inefficient, but that's mostly CPU-wise. At the end of each transaction
the dict is ready to be garbage collected, so there is no memory leak.
It seems likely that the crashing is due to the entire load on the
memory system (huge ets tables?) and perhaps a leak elsewhere.

    /Richard