<div dir="ltr"><div><div><div>Our Erlang server looks have a serious memory leak, the VM memory usage is low but top is high.<br><br><b># Env</b><br><br>Erlang: R16B02 <br>OS: Ubuntu 12.04.5 LTS \n \l   X86_64<br><br></div><b># Server Architecture<br><br></b></div>1. RabbitMQ client consumer Message  <br>2. MQ client cast message to gen_server receiver <br>3. receiver cast message to worker pool (managed by poolboy)<br><div><br><b># Erlang VM</b><br><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div>    > erlang:memory().<br>    [{total,424544992},<br>     {processes,293961840},<br>     {processes_used,293937232},<br>     {system,130583152},<br>     {atom,553569},<br>     {atom_used,521929},<br>     {binary,9794704},<br>     {code,14041920},<br>     {ets,5632280}]<br></div></blockquote><div><br><b># Allocated Memory</b><br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">    > recon_alloc:memory(allocated).<br>    2570059776<br>    <br>     recon_alloc:memory(allocated_types).<br>    [{binary_alloc,163577856},<br>     {driver_alloc,11010048},<br>     {eheap_alloc,2165309440},<br>     {ets_alloc,11010048},<br>     {fix_alloc,50855936},<br>     {ll_alloc,156237824},<br>     {sl_alloc,2097152},<br>     {std_alloc,6815744},<br>     {temp_alloc,3145728}]<br></blockquote></div><div><br><b># allocate binary</b><br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">    > recon:bin_leak(5).<br>    [{<0.440.0>,-769,<br>      [{current_function,{gen_server,loop,6}},<br>       {initial_call,{proc_lib,init_p,5}}]},<br>     {<0.446.0>,-230,<br>      [{current_function,{gen_server,loop,6}},<br>       {initial_call,{proc_lib,init_p,5}}]},<br>     {<0.450.0>,-179,<br>      [{current_function,{gen_server,loop,6}},<br>       {initial_call,{proc_lib,init_p,5}}]},<br>     {<0.12497.0>,-147,<br>      [{current_function,{gen,do_call,4}},<br>       {initial_call,{proc_lib,init_p,5}}]},<br>     {<0.434.0>,-145,<br>      [{current_function,{cberl_worker,mget,4}},<br>       {initial_call,{proc_lib,init_p,5}}]}]<br></blockquote><br><br><b># Do garbage</b><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div>    7> erlang:garbage_collect().<br>    true<br>    8> erlang:memory().<br>    [{total,381782256},<br>     {processes,251371752},<br>     {processes_used,251361352},<br>     {system,130410504},<br>     {atom,553569},<br>     {atom_used,521929},<br>     {binary,9230384},<br>     {code,14041920},<br>     {ets,5675528}]<br>    > recon_alloc:memory(allocated_types).<br>    [{binary_alloc,150994944},<br>     {driver_alloc,11010048},<br>     {eheap_alloc,2154823680},<br>     {ets_alloc,11010048},<br>     {fix_alloc,50855936},<br>     {ll_alloc,156237824},<br>     {sl_alloc,2097152},<br>     {std_alloc,6815744},<br>     {temp_alloc,3145728}]<br></div></blockquote><br><b># Fragmentation</b><br><br>I execute recon_alloc:fragmentation(current) and recon_alloc:fragmentation(max), find some allocator current usage is lower than max usage.<br><br><b>## Current usage</b><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">     {{binary_alloc,0},<br>      [{sbcs_usage,1.0},<br>       {mbcs_usage,0.037804497612847224},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,178384},<br>       {mbcs_carriers_size,4718592}]},<br>     {{binary_alloc,2},<br>      [{sbcs_usage,2.0},<br>       {mbcs_usage,0.05326200786389803},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,4775112},<br>       {mbcs_carriers_size,89653248}]},<br>     {{binary_alloc,1},<br>      [{sbcs_usage,2.0},<br>       {mbcs_usage,0.0643930146188447},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,4456384},<br>       {mbcs_carriers_size,69206016}]},<br></blockquote><div><div><div><br><b>## Max usage</b><br><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div>     {{binary_alloc,0},<br>      [{sbcs_usage,1.0},<br>       {mbcs_usage,0.7732696533203125},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,24324960},<br>       {mbcs_carriers_size,31457280}]},<br>     {{binary_alloc,2},<br>      [{sbcs_usage,1.0},<br>       {mbcs_usage,0.938345729714573},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,149064912},<br>       {mbcs_carriers_size,158859264}]},<br>     {{binary_alloc,0},<br>      [{sbcs_usage,1.0},<br>       {mbcs_usage,0.7732696533203125},<br>       {sbcs_block_size,0},<br>       {sbcs_carriers_size,0},<br>       {mbcs_block_size,24324960},<br>       {mbcs_carriers_size,31457280}]},<br></div></blockquote><div><br>Does it mean Erlang server have lots of Memory fragmentation since MQ binary message pass through multi gen_server?<br><br></div><div>How can I move on for this issue?<br><br></div><div>Thanks,<br></div><div>Linbo<br></div></div></div></div>