[erlang-questions] Garbage Collection, BEAM memory and Erlang memory

Roberto Ostinelli roberto@REDACTED
Fri Jan 23 14:33:04 CET 2015


Hi Lucas,
I'm using 17.4 on ubuntu LTS 14.04. The settings I use are:

+K true
+P 2000000
+Q 1000000

Please find here below an additional set of data with the changes I've
stated here above (binary:copy/0 and the fullsweep_after 0 in the router).

Of the recon library I'm using the following calls:

recon_alloc:memory(allocated, current).
recon_alloc:memory(used, current).
recon_alloc:memory(usage, current).


All of this data is taken at random intervals of time.

- BEAM process RES memory:* 2.732 GB*
- Erlang memory:
[{total,1.7860298827290535},
 {processes,1.4158401936292648},
 {processes_used,1.4157484397292137},
 {system,0.37018968909978867},
 {atom,4.000673070549965e-4},
 {atom_used,3.846092149615288e-4},
 {binary,0.20867645740509033},
 {code,0.009268132038414478},
 {ets,0.004821933805942535}]
- recon_alloc:
allocated: 3015796080 (2.808 GB)
used: 2161850416
usage: 0.7187714029859935


- BEAM process RES memory:* 2.813 GB*
- Erlang memory:
[{total,2.026990756392479},
 {processes,1.6270370781421661},
 {processes_used,1.6269719526171684},
 {system,0.3999536782503128},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.23845425993204117},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3098895728 (2.886 GB)
used: 2176172480
usage: 0.7023218278482198


- BEAM process RES memory:* 3.029 GB*
- Erlang memory:
[{total,2.351852521300316},
 {processes,1.9361207410693169},
 {processes_used,1.9360847249627113},
 {system,0.415731780230999},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.2539009377360344},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3337524592 (3.108 GB)
used: 2525365352
usage: 0.7548030747173055


- BEAM process RES memory:* 3.099 GB*
- Erlang memory:
[{total,2.0704088881611824},
 {processes,1.6625376418232918},
 {processes_used,1.6624245047569275},
 {system,0.4078712463378906},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.24636883288621902},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3400623472 (3.167 GB)
used: 2222575336
usage: 0.6552131374790817


- BEAM process RES memory:* 3.132 GB*
- Erlang memory:
[{total,2.367126949131489},
 {processes,1.9388784170150757},
 {processes_used,1.938723236322403},
 {system,0.4282485321164131},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.2667432576417923},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3435644272 (3.200 GB)
used: 2541469864
usage: 0.7397146313173368


- BEAM process RES memory:* 3.307 GB*
- Erlang memory:
[{total,2.379016488790512},
 {processes,1.9780860394239426},
 {processes_used,1.9779272973537445},
 {system,0.4009304493665695},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.23943009227514267},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3619329392 (3.371 GB)
used: 2554804000
usage: 0.704330124232303


- BEAM process RES memory:* 3.351 GB*
- Erlang memory:
[{total,2.607522390782833},
 {processes,2.168950654566288},
 {processes_used,2.1688189953565598},
 {system,0.4385717362165451},
 {atom,4.000673070549965e-4},
 {atom_used,3.8593634963035583e-4},
 {binary,0.2771267145872116},
 {code,0.009311830624938011},
 {ets,0.004802137613296509}]
- recon_alloc:
allocated: 3669321072 (3.417 GB)
used: 2799919616
usage: 0.7629933213977411


- BEAM process RES memory:* 3.469 GB*
- Erlang memory:
[{total,2.2596140429377556},
 {processes,1.8098593652248383},
 {processes_used,1.8098137602210045},
 {system,0.44975467771291733},
 {atom,4.000673070549965e-4},
 {atom_used,3.86836938560009e-4},
 {binary,0.2881493419408798},
 {code,0.009375222958624363},
 {ets,0.00480380654335022}]
- recon_alloc:
allocated: 3789014384 (3.528 GB)
used: 2425613912
usage: 0.6401929098614897


- BEAM process RES memory:* 3.660 GB*
- Erlang memory:
[{total,2.692381367087364},
 {processes,2.2282255738973618},
 {processes_used,2.228082850575447},
 {system,0.46415579319000244},
 {atom,4.000673070549965e-4},
 {atom_used,3.86836938560009e-4},
 {binary,0.30247989296913147},
 {code,0.009375222958624363},
 {ets,0.00480380654335022}]
- recon_alloc:
allocated: 3993933168 (3.719 GB)
used: 2890714704
usage: 0.7233507625681828


- BEAM process RES memory:* 3.667 GB*
- Erlang memory:
[{total,2.4165985733270645},
 {processes,1.9264011159539223},
 {processes_used,1.9263720959424973},
 {system,0.49019745737314224},
 {atom,4.000673070549965e-4},
 {atom_used,3.86836938560009e-4},
 {binary,0.3284950777888298},
 {code,0.009375222958624363},
 {ets,0.00480380654335022}]
- recon_alloc:
allocated: 4001830256 (3.727 GB)
used: 2594872464
usage: 0.6483950197811689


It looks like the memory allocated has some mismatch with the memory
reported by the OS, but maybe this is just a timing issue (since top
provides an average during a period of time)?

Anyway, the BEAM process keeps on increasing. Also, it looks like memory
usage bounces anywhere from 64-72% and I don't know if that's a good figure
or not.

This is what I get from a check on memory fragmentation:

1> recon_alloc:fragmentation(current).
[{{eheap_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6252789717100151},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,391670536},
   {mbcs_carriers_size,626393264}]},
 {{eheap_alloc,2},
  [{sbcs_usage,0.6168021896258503},
   {mbcs_usage,0.6893887270883688},
   {sbcs_block_size,371384},
   {sbcs_carriers_size,602112},
   {mbcs_block_size,370926112},
   {mbcs_carriers_size,538050736}]},
 {{eheap_alloc,3},
  [{sbcs_usage,0.9991333400321544},
   {mbcs_usage,0.7006580932915004},
   {sbcs_block_size,5091008},
   {sbcs_carriers_size,5095424},
   {mbcs_block_size,324091688},
   {mbcs_carriers_size,462553264}]},
 {{eheap_alloc,4},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6924985776876923},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,305976264},
   {mbcs_carriers_size,441843888}]},
 {{eheap_alloc,5},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6397496430493375},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,207536944},
   {mbcs_carriers_size,324403376}]},
 {{eheap_alloc,8},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6660125315617468},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,166472816},
   {mbcs_carriers_size,249954480}]},
 {{eheap_alloc,6},
  [{sbcs_usage,0.9980070153061225},
   {mbcs_usage,0.6770963446791575},
   {sbcs_block_size,600912},
   {sbcs_carriers_size,602112},
   {mbcs_block_size,169065768},
   {mbcs_carriers_size,249692336}]},
 {{eheap_alloc,7},
  [{sbcs_usage,0.997382155987395},
   {mbcs_usage,0.6925225022824623},
   {sbcs_block_size,972296},
   {sbcs_carriers_size,974848},
   {mbcs_block_size,167834424},
   {mbcs_carriers_size,242352304}]},
 {{ll_alloc,0},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7387279228243809},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,112706224},
   {mbcs_carriers_size,152567976}]},
 {{binary_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7614033191804973},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,58507096},
   {mbcs_carriers_size,76841136}]},
 {{binary_alloc,4},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7179048085736076},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,46131288},
   {mbcs_carriers_size,64258224}]},
 {{binary_alloc,7},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6128136272843548},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,27651200},
   {mbcs_carriers_size,45121712}]},
 {{binary_alloc,5},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6492102167567332},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,32186648},
   {mbcs_carriers_size,49578160}]},
 {{binary_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7715637758298944},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,56051664},
   {mbcs_carriers_size,72646832}]},
 {{binary_alloc,3},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7514879308127178},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,49077272},
   {mbcs_carriers_size,65306800}]},
 {{binary_alloc,6},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.6757064168943689},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,25706456},
   {mbcs_carriers_size,38043824}]},
 {{binary_alloc,8},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7016146506167494},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,25956408},
   {mbcs_carriers_size,36995248}]},
 {{fix_alloc,4},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7409146075507547},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,22748888},
   {mbcs_carriers_size,30703792}]},
 {{fix_alloc,2},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.8581522751225302},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,33547232},
   {mbcs_carriers_size,39092400}]},
 {{fix_alloc,3},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.8630094940716118},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,26497664},
   {mbcs_carriers_size,...}]},
 {{driver_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7615924083651237},
   {sbcs_block_size,0},
   {sbcs_carriers_size,0},
   {mbcs_block_size,...},
   {...}]},
 {{fix_alloc,1},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.8947156992151927},
   {sbcs_block_size,0},
   {sbcs_carriers_size,...},
   {...}|...]},
 {{fix_alloc,5},
  [{sbcs_usage,1.0},
   {mbcs_usage,0.7166474590927997},
   {sbcs_block_size,...},
   {...}|...]},
 {{ll_alloc,7},[{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
 {{driver_alloc,2},[{sbcs_usage,...},{...}|...]},
 {{driver_alloc,4},[{...}|...]},
 {{ll_alloc,...},[...]},
 {{...},...},
 {...}|...]


Unless I'm mistaken reading this data, everything looks fine there, with
normal usages.

What can I do else to address this issue?

Thank you for your help.
r.




On Fri, Jan 23, 2015 at 10:40 AM, Lukas Larsson <lukas@REDACTED>
wrote:

> Hello Roberto,
>
> I cannot find any information about which version of Erlang/OTP that you
> are running, nor which setting you have given to VM at startup?
>
> On Thu, Jan 22, 2015 at 5:33 PM, Roberto Ostinelli <roberto@REDACTED>
> wrote:
>
>>
>> I get my erlang memory with this simple call (I just convert everything
>> to GB, thanks to Ferd and his article
>> https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole):
>>
>>
> Have you also read the part about the memory fragmentation issues that Mr
> Hebert described? Specifically running
> recon_alloc:memory(allocated,current) and recon_alloc:memory(usage,
> current) and checking those values during the test.
>
> Lukas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150123/1906c457/attachment.htm>


More information about the erlang-questions mailing list