[erlang-questions] Erlang OTP 17.3 memory fragment problem (maybe)

[Daniel]

> On Apr 13, 2015, at 3:28 PM, Lukas Larsson <garazdawi@REDACTED> wrote:
> 
> Hello Daniel,
> 
> On Mon, Apr 13, 2015 at 3:13 AM, Daniel <liudanking@
> 
> So my question is:
> Does it means that  I meat a ‘classic' memory fragmentation problem? If YES, how can I free the fragment memory to OS?Because my ejabberd have been killed by Linux OOM for several times. If NO, what cause the memory increase all the time?
> 
> Would you mind checking how much of the binary_alloc memory is part of the mbc pool? ("erlang:system_info({allocator,binary_alloc}).", look for mbcs_pool). If it is very large you may have hit the bug described in OTP-12323, which has been fixed in Erlang/OTP 17.4. The scenario you are describing where the memory utilization constantly decreases sound very similar to what the bug fixed. If you don't want to upgrade to 17.4, you can also disable the pool in 17.3.
> 
> Lukas

Hi, Dear Lukas and list,

Currently, linux *top* shows that beam.smp reserves 6.5GB memory , recon_alloc:memory(allocated) outputs 6.5GB and recon_alloc:memory(usage) outputs 0.0969.

Bellow is my binary_alloc memory of mbcs_pool:

(ejabberd@REDACTED)8> erlang:system_info({allocator,binary_alloc}).
[{instance,0,
           [{versions,"0.9","3.0"},
            {options,[{e,true},
                      {t,true},
                      {ramv,false},
                      {sbct,524288},
                      {asbcst,4145152},
                      {rsbcst,20},
                      {rsbcmt,80},
                      {rmbcmt,50},
                      {mmbcs,65536},
                      {mmmbc,18446744073709551615},
                      {mmsbc,256},
                      {lmbcs,5242880},
                      {smbcs,524288},
                      {mbcgs,10},
                      {acul,0},
                      {as,aoffcbf}]},
            {mbcs,[{blocks,4,4,95},
                   {blocks_size,84208,84208,607944},
                   {carriers,2,2,3},
                   {mseg_alloc_carriers,1},
                   {sys_alloc_carriers,1},
                   {carriers_size,589984,589984,1638560},
                   {mseg_alloc_carriers_size,524288},
                   {sys_alloc_carriers_size,65696}]},
            {sbcs,[{blocks,0,0,0},
                   {blocks_size,0,0,0},
                   {carriers,0,0,0},
                   {mseg_alloc_carriers,0},
                   {sys_alloc_carriers,0},
                   {carriers_size,0,0,0},
                   {mseg_alloc_carriers_size,0},
                   {sys_alloc_carriers_size,0}]},
            {calls,[{binary_alloc,0,11582},
                    {binary_free,0,11578},
                    {binary_realloc,0,0},
                    {mseg_alloc,0,23},
                    {mseg_dealloc,0,22},
                    {mseg_realloc,0,0},
                    {sys_alloc,0,1},
                    {sys_free,0,0},
                    {sys_realloc,0,0}]}]},
 {instance,1,
           [{versions,"0.9","3.0"},
            {options,[{e,true},
                      {t,true},
                      {ramv,false},
                      {sbct,524288},
                      {asbcst,4145152},
                      {rsbcst,20},
                      {rsbcmt,80},
                      {rmbcmt,50},
                      {mmbcs,65536},
                      {mmmbc,18446744073709551615},
                      {mmsbc,256},
                      {lmbcs,5242880},
                      {smbcs,524288},
                      {mbcgs,10},
                      {acul,60},
                      {as,aoffcbf}]},
            {mbcs,[{blocks,91398,118957,1072611},
                   {blocks_size,19822160,25965328,154167640},
                   {carriers,7,10,29},
                   {mseg_alloc_carriers,6},
                   {sys_alloc_carriers,1},
                   {carriers_size,29425824,46203040,205586592},
                   {mseg_alloc_carriers_size,29360128},
                   {sys_alloc_carriers_size,65696}]},
            {mbcs_pool,[{blocks,52044},
                        {blocks_size,22538272},
                        {carriers,115},
                        {carriers_size,792723456}]},
            {sbcs,[{blocks,0,0,13},
                   {blocks_size,0,0,4312368},
                   {carriers,0,0,13},
                   {mseg_alloc_carriers,0},
                   {sys_alloc_carriers,0},
                   {carriers_size,0,0,7483392},
                   {mseg_alloc_carriers_size,0},
                   {sys_alloc_carriers_size,0}]},
            {calls,[{binary_alloc,1,808658319},
                    {binary_free,1,808237034},
                    {binary_realloc,0,660820827},
                    {mseg_alloc,0,25140},
                    {mseg_dealloc,0,24698},
                    {mseg_realloc,0,22983},
                    {sys_alloc,0,1},
                    {sys_free,0,0},
                    {sys_realloc,0,0}]}]},
 {instance,2,
           [{versions,"0.9","3.0"},
            {options,[{e,true},
                      {t,true},
                      {ramv,false},
                      {sbct,524288},
                      {asbcst,4145152},
                      {rsbcst,20},
                      {rsbcmt,80},
                      {rmbcmt,50},
                      {mmbcs,65536},
                      {mmmbc,18446744073709551615},
                      {mmsbc,256},
                      {lmbcs,5242880},
                      {smbcs,524288},
                      {mbcgs,10},
                      {acul,60},
                      {as,aoffcbf}]},
            {mbcs,[{blocks,16137,16819,1068983},
                   {blocks_size,5291776,5983792,151274536},
                   {carriers,4,4,28},
                   {mseg_alloc_carriers,3},
                   {sys_alloc_carriers,1},
                   {carriers_size,13697184,13697184,198246560},
                   {mseg_alloc_carriers_size,13631488},
                   {sys_alloc_carriers_size,65696}]},
            {mbcs_pool,[{blocks,139776},
                        {blocks_size,35429768},
                        {carriers,752},
                        {carriers_size,5443158016}]},
            {sbcs,[{blocks,0,0,13},
                   {blocks_size,0,0,4526120},
                   {carriers,0,0,13},
                   {mseg_alloc_carriers,0},
                   {sys_alloc_carriers,0},
                   {carriers_size,0,0,7524352},
                   {mseg_alloc_carriers_size,0},
                   {sys_alloc_carriers_size,0}]},
            {calls,[{binary_alloc,1,757135985},
                    {binary_free,1,757257915},
                    {binary_realloc,0,646773973},
                    {mseg_alloc,0,24913},
                    {mseg_dealloc,0,24479},
                    {mseg_realloc,0,22617},
                    {sys_alloc,0,1},
                    {sys_free,0,0},
                    {sys_realloc,0,0}]}]}]

mbcs_pool cost (5443158016 + 792723456) bytes (5.8GB) memory.