<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>I'm not sure about the crash dump. But if you're able to re-run it, adding in os_mon should be quite helpful. You can set up listeners that alerts when processes ends up using memory. From there, while the system is still running, you can investigate more on what the process is doing - maybe a sys:get_status on the erlang shell.<br><br>Sent from my iPhone</div><div><br>On Mar 17, 2014, at 4:51 PM, József Bérces <<a href="mailto:jozsef.berces@ericsson.com">jozsef.berces@ericsson.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">I receive the classic “eheap_alloc: Cannot allocate…” message. It wants to allocate ~1GB memory and that fails. That is fine, I am doing something wrong. So I took the crash dump and tried to find out which one of my processes is the guilty
one. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Unfortunately, I cannot tell it from the crash dump. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The memory section says:<o:p></o:p></p>
<p class="MsoNormal">=memory<o:p></o:p></p>
<p class="MsoNormal">total: 15447352528<o:p></o:p></p>
<p class="MsoNormal">processes: 15140232809<o:p></o:p></p>
<p class="MsoNormal">processes_used: 15140005610<o:p></o:p></p>
<p class="MsoNormal">system: 307119719<o:p></o:p></p>
<p class="MsoNormal">atom: 512601<o:p></o:p></p>
<p class="MsoNormal">atom_used: 496586<o:p></o:p></p>
<p class="MsoNormal">binary: 148574400<o:p></o:p></p>
<p class="MsoNormal">code: 21228007<o:p></o:p></p>
<p class="MsoNormal">ets: 119988984<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have 16GB RAM, so the processes use almost all. There are 4010 processes. 1 garbing, 31 scheduled, 3978 waiting. If I sum stack+heap of all the processes then I get ~700MB. That is very far from 16GB. Here are the top 10 stack+heap processes:<o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="510" style="width:382.8pt;margin-left:-1.15pt;border-collapse:collapse">
<tbody>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;background:#0070C0;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><b><span style="color:white">Pid <o:p></o:p></span></b></p>
</td>
<td width="146" nowrap="" valign="bottom" style="width:109.65pt;border:solid windowtext 1.0pt;border-left:none;background:#0070C0;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><b><span style="color:white">State<o:p></o:p></span></b></p>
</td>
<td width="96" nowrap="" valign="bottom" style="width:72.3pt;border:solid windowtext 1.0pt;border-left:none;background:#0070C0;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><b><span style="color:white">Reductions<o:p></o:p></span></b></p>
</td>
<td width="85" nowrap="" valign="bottom" style="width:64.05pt;border:solid windowtext 1.0pt;border-left:none;background:#0070C0;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><b><span style="color:white">Stack+heap<o:p></o:p></span></b></p>
</td>
<td width="88" nowrap="" valign="bottom" style="width:66.0pt;border:solid windowtext 1.0pt;border-left:none;background:#0070C0;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><b><span style="color:white">MsgQ Length<o:p></o:p></span></b></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.21060.67><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Garbing (limited info)<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">1,508,838,180<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">145,962,050<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">1<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.25689.27><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">86,670,344<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">145,962,050<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.10003.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">1,363,039<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">38,263,080<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.15943.66><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">1,882,465,380<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">30,610,465<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.15879.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">471,549<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">30,610,465<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.31854.67><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">154,500,777<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">24,488,375<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.16221.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">262,114<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">24,488,375<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.16628.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">117,268<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">24,488,375<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.15878.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">453,490<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">19,590,700<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="94" nowrap="" valign="bottom" style="width:70.8pt;border:solid windowtext 1.0pt;border-top:none;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black"><0.16235.68><o:p></o:p></span></p>
</td>
<td width="146" style="width:109.65pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal"><span style="color:black">Waiting<o:p></o:p></span></p>
</td>
<td width="96" style="width:72.3pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">181,968<o:p></o:p></span></p>
</td>
<td width="85" style="width:64.05pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">19,590,700<o:p></o:p></span></p>
</td>
<td width="88" style="width:66.0pt;border-top:none;border-left:none;border-bottom:solid windowtext 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 5.4pt 0in 5.4pt;height:15.0pt">
<p class="MsoNormal" align="right" style="text-align:right"><span style="color:black">0<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any ideas how to tell which process needs ~1GB memory?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Jozsef<o:p></o:p></p>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>erlang-questions mailing list</span><br><span><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a></span><br><span><a href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a></span><br></div></blockquote></body></html>