[erlang-questions] Garbage Collection, BEAM memory and Erlang memory
Roberto Ostinelli
roberto@REDACTED
Fri Jan 23 14:33:04 CET 2015
Hi Lucas,
I'm using 17.4 on ubuntu LTS 14.04. The settings I use are:
+K true
+P 2000000
+Q 1000000
Please find here below an additional set of data with the changes I've
stated here above (binary:copy/0 and the fullsweep_after 0 in the router).
Of the recon library I'm using the following calls:
recon_alloc:memory(allocated, current).
recon_alloc:memory(used, current).
recon_alloc:memory(usage, current).
All of this data is taken at random intervals of time.
- BEAM process RES memory:* 2.732 GB*
- Erlang memory:
[{total,1.7860298827290535},
{processes,1.4158401936292648},
{processes_used,1.4157484397292137},
{system,0.37018968909978867},
{atom,4.000673070549965e-4},
{atom_used,3.846092149615288e-4},
{binary,0.20867645740509033},
{code,0.009268132038414478},
{ets,0.004821933805942535}]
- recon_alloc:
allocated: 3015796080 (2.808 GB)
used: 2161850416
usage: 0.7187714029859935
- BEAM process RES memory:* 2.813 GB*
- Erlang memory:
[{total,2.026990756392479},
{processes,1.6270370781421661},
{processes_used,1.6269719526171684},
{system,0.3999536782503128},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.23845425993204117},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3098895728 (2.886 GB)
used: 2176172480
usage: 0.7023218278482198
- BEAM process RES memory:* 3.029 GB*
- Erlang memory:
[{total,2.351852521300316},
{processes,1.9361207410693169},
{processes_used,1.9360847249627113},
{system,0.415731780230999},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.2539009377360344},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3337524592 (3.108 GB)
used: 2525365352
usage: 0.7548030747173055
- BEAM process RES memory:* 3.099 GB*
- Erlang memory:
[{total,2.0704088881611824},
{processes,1.6625376418232918},
{processes_used,1.6624245047569275},
{system,0.4078712463378906},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.24636883288621902},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3400623472 (3.167 GB)
used: 2222575336
usage: 0.6552131374790817
- BEAM process RES memory:* 3.132 GB*
- Erlang memory:
[{total,2.367126949131489},
{processes,1.9388784170150757},
{processes_used,1.938723236322403},
{system,0.4282485321164131},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.2667432576417923},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3435644272 (3.200 GB)
used: 2541469864
usage: 0.7397146313173368
- BEAM process RES memory:* 3.307 GB*
- Erlang memory:
[{total,2.379016488790512},
{processes,1.9780860394239426},
{processes_used,1.9779272973537445},
{system,0.4009304493665695},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.23943009227514267},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3619329392 (3.371 GB)
used: 2554804000
usage: 0.704330124232303
- BEAM process RES memory:* 3.351 GB*
- Erlang memory:
[{total,2.607522390782833},
{processes,2.168950654566288},
{processes_used,2.1688189953565598},
{system,0.4385717362165451},
{atom,4.000673070549965e-4},
{atom_used,3.8593634963035583e-4},
{binary,0.2771267145872116},
{code,0.009311830624938011},
{ets,0.004802137613296509}]
- recon_alloc:
allocated: 3669321072 (3.417 GB)
used: 2799919616
usage: 0.7629933213977411
- BEAM process RES memory:* 3.469 GB*
- Erlang memory:
[{total,2.2596140429377556},
{processes,1.8098593652248383},
{processes_used,1.8098137602210045},
{system,0.44975467771291733},
{atom,4.000673070549965e-4},
{atom_used,3.86836938560009e-4},
{binary,0.2881493419408798},
{code,0.009375222958624363},
{ets,0.00480380654335022}]
- recon_alloc:
allocated: 3789014384 (3.528 GB)
used: 2425613912
usage: 0.6401929098614897
- BEAM process RES memory:* 3.660 GB*
- Erlang memory:
[{total,2.692381367087364},
{processes,2.2282255738973618},
{processes_used,2.228082850575447},
{system,0.46415579319000244},
{atom,4.000673070549965e-4},
{atom_used,3.86836938560009e-4},
{binary,0.30247989296913147},
{code,0.009375222958624363},
{ets,0.00480380654335022}]
- recon_alloc:
allocated: 3993933168 (3.719 GB)
used: 2890714704
usage: 0.7233507625681828
- BEAM process RES memory:* 3.667 GB*
- Erlang memory:
[{total,2.4165985733270645},
{processes,1.9264011159539223},
{processes_used,1.9263720959424973},
{system,0.49019745737314224},
{atom,4.000673070549965e-4},
{atom_used,3.86836938560009e-4},
{binary,0.3284950777888298},
{code,0.009375222958624363},
{ets,0.00480380654335022}]
- recon_alloc:
allocated: 4001830256 (3.727 GB)
used: 2594872464
usage: 0.6483950197811689
It looks like the memory allocated has some mismatch with the memory
reported by the OS, but maybe this is just a timing issue (since top
provides an average during a period of time)?
Anyway, the BEAM process keeps on increasing. Also, it looks like memory
usage bounces anywhere from 64-72% and I don't know if that's a good figure
or not.
This is what I get from a check on memory fragmentation:
1> recon_alloc:fragmentation(current).
[{{eheap_alloc,1},
[{sbcs_usage,1.0},
{mbcs_usage,0.6252789717100151},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,391670536},
{mbcs_carriers_size,626393264}]},
{{eheap_alloc,2},
[{sbcs_usage,0.6168021896258503},
{mbcs_usage,0.6893887270883688},
{sbcs_block_size,371384},
{sbcs_carriers_size,602112},
{mbcs_block_size,370926112},
{mbcs_carriers_size,538050736}]},
{{eheap_alloc,3},
[{sbcs_usage,0.9991333400321544},
{mbcs_usage,0.7006580932915004},
{sbcs_block_size,5091008},
{sbcs_carriers_size,5095424},
{mbcs_block_size,324091688},
{mbcs_carriers_size,462553264}]},
{{eheap_alloc,4},
[{sbcs_usage,1.0},
{mbcs_usage,0.6924985776876923},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,305976264},
{mbcs_carriers_size,441843888}]},
{{eheap_alloc,5},
[{sbcs_usage,1.0},
{mbcs_usage,0.6397496430493375},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,207536944},
{mbcs_carriers_size,324403376}]},
{{eheap_alloc,8},
[{sbcs_usage,1.0},
{mbcs_usage,0.6660125315617468},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,166472816},
{mbcs_carriers_size,249954480}]},
{{eheap_alloc,6},
[{sbcs_usage,0.9980070153061225},
{mbcs_usage,0.6770963446791575},
{sbcs_block_size,600912},
{sbcs_carriers_size,602112},
{mbcs_block_size,169065768},
{mbcs_carriers_size,249692336}]},
{{eheap_alloc,7},
[{sbcs_usage,0.997382155987395},
{mbcs_usage,0.6925225022824623},
{sbcs_block_size,972296},
{sbcs_carriers_size,974848},
{mbcs_block_size,167834424},
{mbcs_carriers_size,242352304}]},
{{ll_alloc,0},
[{sbcs_usage,1.0},
{mbcs_usage,0.7387279228243809},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,112706224},
{mbcs_carriers_size,152567976}]},
{{binary_alloc,1},
[{sbcs_usage,1.0},
{mbcs_usage,0.7614033191804973},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,58507096},
{mbcs_carriers_size,76841136}]},
{{binary_alloc,4},
[{sbcs_usage,1.0},
{mbcs_usage,0.7179048085736076},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,46131288},
{mbcs_carriers_size,64258224}]},
{{binary_alloc,7},
[{sbcs_usage,1.0},
{mbcs_usage,0.6128136272843548},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,27651200},
{mbcs_carriers_size,45121712}]},
{{binary_alloc,5},
[{sbcs_usage,1.0},
{mbcs_usage,0.6492102167567332},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,32186648},
{mbcs_carriers_size,49578160}]},
{{binary_alloc,2},
[{sbcs_usage,1.0},
{mbcs_usage,0.7715637758298944},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,56051664},
{mbcs_carriers_size,72646832}]},
{{binary_alloc,3},
[{sbcs_usage,1.0},
{mbcs_usage,0.7514879308127178},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,49077272},
{mbcs_carriers_size,65306800}]},
{{binary_alloc,6},
[{sbcs_usage,1.0},
{mbcs_usage,0.6757064168943689},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,25706456},
{mbcs_carriers_size,38043824}]},
{{binary_alloc,8},
[{sbcs_usage,1.0},
{mbcs_usage,0.7016146506167494},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,25956408},
{mbcs_carriers_size,36995248}]},
{{fix_alloc,4},
[{sbcs_usage,1.0},
{mbcs_usage,0.7409146075507547},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,22748888},
{mbcs_carriers_size,30703792}]},
{{fix_alloc,2},
[{sbcs_usage,1.0},
{mbcs_usage,0.8581522751225302},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,33547232},
{mbcs_carriers_size,39092400}]},
{{fix_alloc,3},
[{sbcs_usage,1.0},
{mbcs_usage,0.8630094940716118},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,26497664},
{mbcs_carriers_size,...}]},
{{driver_alloc,1},
[{sbcs_usage,1.0},
{mbcs_usage,0.7615924083651237},
{sbcs_block_size,0},
{sbcs_carriers_size,0},
{mbcs_block_size,...},
{...}]},
{{fix_alloc,1},
[{sbcs_usage,1.0},
{mbcs_usage,0.8947156992151927},
{sbcs_block_size,0},
{sbcs_carriers_size,...},
{...}|...]},
{{fix_alloc,5},
[{sbcs_usage,1.0},
{mbcs_usage,0.7166474590927997},
{sbcs_block_size,...},
{...}|...]},
{{ll_alloc,7},[{sbcs_usage,1.0},{mbcs_usage,...},{...}|...]},
{{driver_alloc,2},[{sbcs_usage,...},{...}|...]},
{{driver_alloc,4},[{...}|...]},
{{ll_alloc,...},[...]},
{{...},...},
{...}|...]
Unless I'm mistaken reading this data, everything looks fine there, with
normal usages.
What can I do else to address this issue?
Thank you for your help.
r.
On Fri, Jan 23, 2015 at 10:40 AM, Lukas Larsson <lukas@REDACTED>
wrote:
> Hello Roberto,
>
> I cannot find any information about which version of Erlang/OTP that you
> are running, nor which setting you have given to VM at startup?
>
> On Thu, Jan 22, 2015 at 5:33 PM, Roberto Ostinelli <roberto@REDACTED>
> wrote:
>
>>
>> I get my erlang memory with this simple call (I just convert everything
>> to GB, thanks to Ferd and his article
>> https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole):
>>
>>
> Have you also read the part about the memory fragmentation issues that Mr
> Hebert described? Specifically running
> recon_alloc:memory(allocated,current) and recon_alloc:memory(usage,
> current) and checking those values during the test.
>
> Lukas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150123/1906c457/attachment.htm>
More information about the erlang-questions
mailing list