<div dir="ltr">Dear List,<div>I'm having some troubles in pinpointing why a node is crashing due to memory issues.</div><div>For info, when it crashes, it does not produce a crash dump. However I've monitored live and I've seen the .beam process eat up all memory until it abruptly exits.<br></div><div><br></div><div>The system is a big router that relays data coming from TCP connections, into other TCP connections.</div><div>I'm using cowboy as the HTTP server that initiates the long-lived TCP connections.</div><div><br></div><div>I've done all the obvious:</div><div><ul><li>Checked the States of my gen_servers and processes.</li><li>Checked my processes mailboxes (the ones with the longest queue have 1 item in the inbox).</li><li>My ETS table memory is constant (see below).</li></ul><div>I put the system under controlled load, and I can see with <font face="monospace, monospace">length(processes()).</font> that my process count is stable, always around 120,000.</div></div><div><br></div><div>I check the processes that are using most memory with this call:</div><div><br></div><div><div><font face="monospace, monospace">MostMemory = fun(N) -></font></div><div><font face="monospace, monospace"> lists:sublist(</font></div><div><font face="monospace, monospace"> lists:sort(</font></div><div><font face="monospace, monospace"> fun({_, _, V1}, {_, _, V2}) -> V1 >= V2 end, </font></div><div><font face="monospace, monospace"> [try</font></div><div><font face="monospace, monospace"> [{memory, Mem}, {registered_name, RegName}] = erlang:process_info(Pid, [memory, registered_name]),</font></div><div><font face="monospace, monospace"> {Pid, RegName, Mem}</font></div><div><font face="monospace, monospace"> catch _:_ -></font></div><div><font face="monospace, monospace"> {Pid, undefined, 0}</font></div><div><font face="monospace, monospace"> end || Pid <- processes(), Pid =/= self()]</font></div><div><font face="monospace, monospace"> ), N)</font></div><div><font face="monospace, monospace"> end.</font></div></div><div><br></div><div>Which always returns very similar numbers:</div><div><br></div><div><font face="monospace, monospace">1> MostMemory(20).</font></div><div><div><font face="monospace, monospace">[{<0.96.0>,[],5180448},</font></div><div><font face="monospace, monospace"> {<0.78.0>,tls_connection_sup,4525096},</font></div><div><font face="monospace, monospace"> {<0.6.0>,error_logger,743776},</font></div><div><font face="monospace, monospace"> {<0.7.0>,application_controller,372592},</font></div><div><font face="monospace, monospace"> {<0.77.0>,ssl_manager,284640},</font></div><div><font face="monospace, monospace"> {<0.11.0>,kernel_sup,176712},</font></div><div><font face="monospace, monospace"> {<0.26.0>,code_server,176272},</font></div><div><font face="monospace, monospace"> {<0.33.0>,[],143064},</font></div><div><font face="monospace, monospace"> {<0.419.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.420.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.421.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.422.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.423.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.424.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.425.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.426.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.427.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.428.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.429.0>,[],142896},</font></div><div><font face="monospace, monospace"> {<0.430.0>,[],142896}]</font></div></div><div><br></div><div>See the last processes there with all identical memory? These are the processes handling the connections, and they stay stable with the same identical number throughout all test.</div><div><br></div><div>I get the pid of the .beam process, and I check its reported RES memory with <font face="monospace, monospace">top -p beam-pid-here</font>.<br></div>
<div>I get my erlang memory with this simple call (I just convert everything to GB, thanks to Ferd and his article <a href="https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole">https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole</a>):</div><div><br></div><div><font face="monospace, monospace">[{K,V / math:pow(1024,3)} || {K,V} <- erlang:memory()].</font><br></div><div><br></div><div>This is what I get (at random time intervals):</div><div><br></div><div>- BEAM process RES memory:<b> 2.751 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.11871287971735},</font></div><div><font face="monospace, monospace"> {processes,1.6582859307527542},</font></div><div><font face="monospace, monospace"> {processes_used,1.6581560596823692},</font></div><div><font face="monospace, monospace"> {system,0.4604269489645958},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.29880597442388535},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004808835685253143}]</font></div></div><div><br></div><div><div>- BEAM process RES memory:<b> 3.039 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.2570599243044853},</font></div><div><font face="monospace, monospace"> {processes,1.7243007272481918},</font></div><div><font face="monospace, monospace"> {processes_used,1.7241046279668808},</font></div><div><font face="monospace, monospace"> {system,0.5327591970562935},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.37129393219947815},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004808835685253143}]</font></div></div></div><div><font face="monospace, monospace"><br></font></div><div><div>- BEAM process RES memory:<b> 3.630 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.677028402686119},</font></div><div><font face="monospace, monospace"> {processes,2.1421403884887695},</font></div><div><font face="monospace, monospace"> {processes_used,2.142106533050537},</font></div><div><font face="monospace, monospace"> {system,0.5348880141973495},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.37329262495040894},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004808835685253143}]</font></div></div></div><div><div><br class="">- BEAM process RES memory:<b> 3.807 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.9233806803822517},</font></div><div><font face="monospace, monospace"> {processes,2.277688652276993},</font></div><div><font face="monospace, monospace"> {processes_used,2.277618482708931},</font></div><div><font face="monospace, monospace"> {system,0.6456920281052589},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.48407071083784103},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004808835685253143}]</font></div><div><br class=""><br class="">- BEAM process RES memory:<b> 4.026 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.8762372359633446},</font></div><div><font face="monospace, monospace"> {processes,2.100425034761429},</font></div><div><font face="monospace, monospace"> {processes_used,2.1003194376826286},</font></div><div><font face="monospace, monospace"> {system,0.7758122012019157},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.6143399104475975},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004808835685253143}]</font></div></div></div></div><div><font face="monospace, monospace"><br></font></div><div><div><br class="">- BEAM process RES memory:<b> 4.136 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.9030912443995476},</font></div><div><font face="monospace, monospace"> {processes,2.028559662401676},</font></div><div><font face="monospace, monospace"> {processes_used,2.0283572375774384},</font></div><div><font face="monospace, monospace"> {system,0.8745315819978714},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.847004845738411e-4},</font></div><div><font face="monospace, monospace"> {binary,0.7129654437303543},</font></div><div><font face="monospace, monospace"> {code,0.00929991528391838},</font></div><div><font face="monospace, monospace"> {ets,0.004809550940990448}]</font></div></div></div><div><font face="monospace, monospace"><br></font></div><div><div><br class="">- BEAM process RES memory:<b> 4.222 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,2.785604253411293},</font></div><div><font face="monospace, monospace"> {processes,1.875294029712677},</font></div><div><font face="monospace, monospace"> {processes_used,1.8752291351556778},</font></div><div><font face="monospace, monospace"> {system,0.910310223698616},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.847004845738411e-4},</font></div><div><font face="monospace, monospace"> {binary,0.7487552836537361},</font></div><div><font face="monospace, monospace"> {code,0.00929991528391838},</font></div><div><font face="monospace, monospace"> {ets,0.004809550940990448}]</font></div></div></div><div><font face="monospace, monospace"><br></font></div><div><br></div><div>As you can see, at the beginning both the BEAM RES memory and the total Erlang memory increase, but after a while it becomes clear that the BEAM process memory keeps increasing while the memory reported as used by Erlang stabilizes, and even decreases.<br></div><div>Erlang reported memory never surpasses 3 GB.</div><div><br></div><div>At this point I tried forcing a Garbage Collection:</div><div><br></div><div><font face="monospace, monospace">[erlang:garbage_collect(Pid) || Pid <- processes()]<br></font></div><div><br></div><div>After that, we went back to:</div><div><br></div><div><div>- BEAM process RES memory:<b> 3.336 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,1.9107630401849747},</font></div><div><font face="monospace, monospace"> {processes,1.5669479593634605},</font></div><div><font face="monospace, monospace"> {processes_used,1.5668926388025284},</font></div><div><font face="monospace, monospace"> {system,0.34381508082151413},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.847004845738411e-4},</font></div><div><font face="monospace, monospace"> {binary,0.18235664814710617},</font></div><div><font face="monospace, monospace"> {code,0.00929991528391838},</font></div><div><font face="monospace, monospace"> {ets,0.004809550940990448}]</font></div></div></div><div><font face="monospace, monospace"><br></font></div><div>However after that, I let the system go and it kept on having the same behavior (and increasing the BEAM memory).</div><div><br></div><div>What puzzles me is that you can clearly see that:</div><div><ul><li>The total memory used by processes is increasing, however the top processes always use the same amount of memory (and the process count is always stable).<br></li><li>Binary consumption also increases, but in proportion with process memory (and my data is <64K so I don't anticipate it being an issue of Refc-binaries not being garbage collected). </li></ul><div>I already hibernate most of the long-term open connections.</div></div><div>I also added a periodic garbage collector on the main router, since it touches all the binaries that go through it, to ensure that all Refc-binaries that hold a reference to the router are garbage collected.</div><div><br></div><div>So I tried the hard approach, and I've set <font face="monospace, monospace">fullsweep_after</font> to 0 as a system flag (passed in as an environment variable <font face="monospace, monospace">-env ERL_FULLSWEEP_AFTER 0</font>).</div><div><br></div><div>After this, I could see notable improvements:</div><div><div><br class="">- BEAM process RES memory:<b> 2.049 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,1.597476489841938},</font></div><div><font face="monospace, monospace"> {processes,1.2037805244326591},</font></div><div><font face="monospace, monospace"> {processes_used,1.2036690935492516},</font></div><div><font face="monospace, monospace"> {system,0.39369596540927887},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.2321353331208229},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004821933805942535}]</font></div><div><br class="">- BEAM process RES memory:<b> 1.919 GB</b></div><div>- Erlang memory:<br></div><div><div><div><font face="monospace, monospace">[{total,1.549286112189293},</font></div><div><font face="monospace, monospace"> {processes,1.1740453317761421},</font></div><div><font face="monospace, monospace"> {processes_used,1.1739420965313911},</font></div><div><font face="monospace, monospace"> {system,0.3752407804131508},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.2134672999382019},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004821933805942535}]</font></div></div><div><br class="">- BEAM process RES memory:<b> 2.004 GB</b></div><div>- Erlang memory:<br></div><div><div><div><font face="monospace, monospace">[{total,1.6023957282304764},</font></div><div><font face="monospace, monospace"> {processes,1.2192133665084839},</font></div><div><font face="monospace, monospace"> {processes_used,1.219102293252945},</font></div><div><font face="monospace, monospace"> {system,0.3831823617219925},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.22155668586492538},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004821933805942535}]</font></div></div><div><br class="">- BEAM process RES memory:<b> 2.456 GB</b></div><div>- Erlang memory:<br></div><div><div><div><font face="monospace, monospace">[{total,1.7860298827290535},</font></div><div><font face="monospace, monospace"> {processes,1.4158401936292648},</font></div><div><font face="monospace, monospace"> {processes_used,1.4157484397292137},</font></div><div><font face="monospace, monospace"> {system,0.37018968909978867},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.20867645740509033},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004821933805942535}]</font></div></div><div><br class="">- BEAM process RES memory:<b> 2.455 GB</b></div><div>- Erlang memory:<br></div><div><div><font face="monospace, monospace">[{total,1.8919306173920631},</font></div><div><font face="monospace, monospace"> {processes,1.4726912006735802},</font></div><div><font face="monospace, monospace"> {processes_used,1.4726523533463478},</font></div><div><font face="monospace, monospace"> {system,0.41923941671848297},</font></div><div><font face="monospace, monospace"> {atom,4.000673070549965e-4},</font></div><div><font face="monospace, monospace"> {atom_used,3.846092149615288e-4},</font></div><div><font face="monospace, monospace"> {binary,0.25766071677207947},</font></div><div><font face="monospace, monospace"> {code,0.009268132038414478},</font></div><div><font face="monospace, monospace"> {ets,0.004821933805942535}]</font></div></div></div></div></div></div></div><div><br></div><div><br></div><div>However, the down size to this is obviously that the CPU load increased almost of a point.</div><div><br></div><div>I also have a GC "guardian" similar to the one that Fred implemented in Heroku's logplex:</div><div><a href="https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl">https://github.com/heroku/logplex/blob/master/src/logplex_leak.erl</a><br></div><div><br></div><div>But this obviously is a guard, not a solution per se.</div><div><br></div><div>Can anyone give me some pointers on how I can process to identify what is going on?</div><div><br></div><div>Thank you,</div><div>r.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>