[erlang-questions] Possilbe Erlang memory fragmentation

[linbo liao]

Our Erlang server looks have a serious memory leak, the VM memory usage is
low but top is high.

*# Env*

Erlang: R16B02
OS: Ubuntu 12.04.5 LTS \n \l   X86_64



*# Server Architecture*
1. RabbitMQ client consumer Message
2. MQ client cast message to gen_server receiver
3. receiver cast message to worker pool (managed by poolboy)

*# Erlang VM*

    > erlang:memory().
>     [{total,424544992},
>      {processes,293961840},
>      {processes_used,293937232},
>      {system,130583152},
>      {atom,553569},
>      {atom_used,521929},
>      {binary,9794704},
>      {code,14041920},
>      {ets,5632280}]
>

*# Allocated Memory*

    > recon_alloc:memory(allocated).
>     2570059776
>
>      recon_alloc:memory(allocated_types).
>     [{binary_alloc,163577856},
>      {driver_alloc,11010048},
>      {eheap_alloc,2165309440},
>      {ets_alloc,11010048},
>      {fix_alloc,50855936},
>      {ll_alloc,156237824},
>      {sl_alloc,2097152},
>      {std_alloc,6815744},
>      {temp_alloc,3145728}]
>

*# allocate binary*

    > recon:bin_leak(5).
>     [{<0.440.0>,-769,
>       [{current_function,{gen_server,loop,6}},
>        {initial_call,{proc_lib,init_p,5}}]},
>      {<0.446.0>,-230,
>       [{current_function,{gen_server,loop,6}},
>        {initial_call,{proc_lib,init_p,5}}]},
>      {<0.450.0>,-179,
>       [{current_function,{gen_server,loop,6}},
>        {initial_call,{proc_lib,init_p,5}}]},
>      {<0.12497.0>,-147,
>       [{current_function,{gen,do_call,4}},
>        {initial_call,{proc_lib,init_p,5}}]},
>      {<0.434.0>,-145,
>       [{current_function,{cberl_worker,mget,4}},
>        {initial_call,{proc_lib,init_p,5}}]}]
>


*# Do garbage*

>     7> erlang:garbage_collect().
>     true
>     8> erlang:memory().
>     [{total,381782256},
>      {processes,251371752},
>      {processes_used,251361352},
>      {system,130410504},
>      {atom,553569},
>      {atom_used,521929},
>      {binary,9230384},
>      {code,14041920},
>      {ets,5675528}]
>     > recon_alloc:memory(allocated_types).
>     [{binary_alloc,150994944},
>      {driver_alloc,11010048},
>      {eheap_alloc,2154823680},
>      {ets_alloc,11010048},
>      {fix_alloc,50855936},
>      {ll_alloc,156237824},
>      {sl_alloc,2097152},
>      {std_alloc,6815744},
>      {temp_alloc,3145728}]
>

*# Fragmentation*

I execute recon_alloc:fragmentation(current) and
recon_alloc:fragmentation(max), find some allocator current usage is lower
than max usage.

*## Current usage*

>      {{binary_alloc,0},
>       [{sbcs_usage,1.0},
>        {mbcs_usage,0.037804497612847224},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,178384},
>        {mbcs_carriers_size,4718592}]},
>      {{binary_alloc,2},
>       [{sbcs_usage,2.0},
>        {mbcs_usage,0.05326200786389803},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,4775112},
>        {mbcs_carriers_size,89653248}]},
>      {{binary_alloc,1},
>       [{sbcs_usage,2.0},
>        {mbcs_usage,0.0643930146188447},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,4456384},
>        {mbcs_carriers_size,69206016}]},
>

*## Max usage*

     {{binary_alloc,0},
>       [{sbcs_usage,1.0},
>        {mbcs_usage,0.7732696533203125},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,24324960},
>        {mbcs_carriers_size,31457280}]},
>      {{binary_alloc,2},
>       [{sbcs_usage,1.0},
>        {mbcs_usage,0.938345729714573},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,149064912},
>        {mbcs_carriers_size,158859264}]},
>      {{binary_alloc,0},
>       [{sbcs_usage,1.0},
>        {mbcs_usage,0.7732696533203125},
>        {sbcs_block_size,0},
>        {sbcs_carriers_size,0},
>        {mbcs_block_size,24324960},
>        {mbcs_carriers_size,31457280}]},
>

Does it mean Erlang server have lots of Memory fragmentation since MQ
binary message pass through multi gen_server?

How can I move on for this issue?

Thanks,
Linbo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20151207/0f86784f/attachment.htm>