[erlang-questions] Using ETS for large amounts of data?

[Ryan Zezeski]

On Sat, Aug 28, 2010 at 2:41 AM, <jay@REDACTED> wrote:

>
> If you use and ETS table, you have to use integer index offsets as the
> return value or the binary data will be copied into the ETS on insert and
> out again on every lookup match.


I think the documentation needs to be updated because I don't witness this
behavior in R14A.  For example, I can read a 436M CSV file, store it in ETS
and not see my memory usage go up.  Furthermore, I can then use lookup and
match 400MB of the binary and also not see the memory go up.

My erl session looked something like:

1> {ok,Data} = file:read_file("436MB_file").
2> ets:new(test,[named_table]).
3> ets:insert(test,{data,Data}).
4> [{data,<<Part:419430400/binary,_/binary>>}] = ets:lookup(test,data).

After line 1, the OS process was using ~445MB of RAM.  After line 4 it was
still using only 445MB and I never saw it move.

-Ryan