[erlang-questions] dealing with large dicts

[Jacob Perkins]

Hey all, I've been having an issue with erlang crashing due to an
out-of-memory error. I'm pretty sure the reason is that I'm looking up,
updating, then writing back a fairly large dict in a mnesia transaction, and
doing that many times for many different dicts. Each dict may contain
thousands of 30 character string keys along with float values. My rough
calculations say that each dict may be on the order of a couple megabytes,
maybe approaching 10M given the indexing overhead of the dict structure. And
since all the value copying happens within a transaction, there's a lot of
memory being used. Here's the basic transaction function I'm doing:
T = fun(Term, Key) ->
         [Dict1] = mnesia:read(Table, Term),
         Dict2 = dict:store(Key, Val, Dict1),
         ok = write_val(Table, Term, Dict2)
     end.

Any thoughts, alternative approaches? Because of erlang's single assignment,
I'm thinking maybe what I need is a kind of "binary dict", something where
the whole structure is not copied when a small part changes, and still
provides very quick value lookup and key iteration. Does this sound
feasible, any thoughts on how to implement it? Is there any existing code
for something like this? Could I still store the binary in mnesia?

Thanks,
Jacob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080904/10c059a8/attachment.htm>