[erlang-questions] binary, ets and memory...

Dmitry Kolesnikov <>
Fri Oct 12 21:15:21 CEST 2012


Hello,

Recently, my system starts to swap. The investigation has indicated that memory consumption was almost twice more then I've expected and to be honest, I've confused why it so...

I am talking about Erlang R15B (erts-5.9) [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

So, I do have two processes 
 * first process handles tcp/ip socket I/O. received data is pushed to second process
 * second process splits binaries binary:split(Buf, [<<$,>>], [global]), parses data and makes list of tuple. When list of tuples is ready, it folds tuples into ets table
    ets:new(cache, [named_table, ordered_set, public]),
    lists:foldl(
       fun({A, B}, Acc) -> ets:insert(cache, {A, B})  end,
       true,
       List
    ).

I do have about 6M tuples, where first element is SHA1 signature, second element is integer. Receiver process pushes 100 tuples per time. I hope you got rough idea.

When cache is populated, I do have following memory usage, it looks suspicious for me:
{total,1179235080},
{processes,2373638},
{processes_used,2373570},
{system,1176861442},
{atom,264505},
{atom_used,253241},
{binary,434761000},   <-- this looks strange for me. Why binaries are left in heap and in ets?
{code,6521469},
{ets,732409416}

If I change my implementation to 
lists:foldl(
       fun({A, B}, Acc) -> ets:insert(cache, {binary:copy(A), B})  end,
       true,
       List
    ).
then memory utilization is on the par with my estimates
{total,701448856},
{processes,2405251},
{processes_used,2405170},
{system,699043605},
{atom,264505},
{atom_used,253241},
{binary,2686280},
{code,6521493},
{ets,686663080}

Best Regards, 
Dmitry
  



More information about the erlang-questions mailing list