[erlang-questions] binary, ets and memory...

[Dmitry Kolesnikov]

Hello,

Recently, my system starts to swap. The investigation has indicated that memory consumption was almost twice more then I've expected and to be honest, I've confused why it so...

I am talking about Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

So, I do have two processes 
 * first process handles tcp/ip socket I/O. received data is pushed to second process
 * second process splits binaries binary:split(Buf, [<<$,>>], [global]), parses data and makes list of tuple. When list of tuples is ready, it folds tuples into ets table
    ets:new(cache, [named_table, ordered_set, public]),
    lists:foldl(
       fun({A, B}, Acc) -> ets:insert(cache, {A, B})  end,
       true,
       List
    ).

I do have about 6M tuples, where first element is SHA1 signature, second element is integer. Receiver process pushes 100 tuples per time. I hope you got rough idea.

When cache is populated, I do have following memory usage, it looks suspicious for me:
{total,1179235080},
{processes,2373638},
{processes_used,2373570},
{system,1176861442},
{atom,264505},
{atom_used,253241},
{binary,434761000},   <-- this looks strange for me. Why binaries are left in heap and in ets?
{code,6521469},
{ets,732409416}

If I change my implementation to 
lists:foldl(
       fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B})  end,
       true,
       List
    ).
then memory utilization is on the par with my estimates
{total,701448856},
{processes,2405251},
{processes_used,2405170},
{system,699043605},
{atom,264505},
{atom_used,253241},
{binary,2686280},
{code,6521493},
{ets,686663080}

Best Regards, 
Dmitry