[erlang-questions] dealing with large dicts

[Richard Carlsson]

Paul Fisher wrote:
> While not always the right thing because of the extra cpu overhead, you 
> can reduce some of the memory overhead and copying with something like this:
> 
> T = fun(Term, Key) ->
>          [CDict1] = mnesia:read(Table, Term),
>          Dict1 = binary_to_term( CDict1 ),
>          Dict2 = dict:store(Key, Val, Dict1),
>          CDict2 = term_to_binary( Dict2, [compressed] ),
>          ok = write_val(Table, Term, CDict2)
>      end.
> 
> This will keep the size down for the copies and should be ref counted 
> when passed from/to mnesia/ets/dets process to retrieve/store.

No. First, term_to_binary and binary_to_term will do exactly as much
work as the original copying. Second, the binaries will use more space
than the internal representation (look at term_to_binary({foo,foo}) and
you'll see why). Third, if mnesia uses dets tables, you'll now have
twice the copying.

The solution to the problem is not to have a dynamic number of tables,
but to use a more traditional database management approach.

    /Richard