<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The old instrumentation module couldn’t trace allocations, and your method for finding (long-lived) binary leaks required a debugger and manual interpretation of block contents which goes far beyond what we can support. Still, there’s nothing
preventing you from doing that with the new instrumentation and I provided brief instructions on how to create the same kind of memory map using `gdb` in the thread you linked. I can’t see why you insist on saying it’s impossible, it’s merely different.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In either case, the problem at hand is transient so going through the blocks wouldn’t help and OP said as much in their first post. We’ll look into adding tracing of some sort for large allocations, be it `LTTng` probes or otherwise.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">/John<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> erlang-questions <erlang-questions-bounces@erlang.org>
<b>On Behalf Of </b>Vans S<br>
<b>Sent:</b> Thursday, April 09, 2020 03:03<br>
<b>To:</b> Erlang (E-mail) <erlang-questions@erlang.org>; Dániel Szoboszlay <dszoboszlay@gmail.com><br>
<b>Subject:</b> Re: Tracing large binary allocations<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">This used to be possible via the instrument module but the functionality got removed in OTP21, I used this functionality to detect a large binheap leak in jiffy way back
when. Theres a lengthy discussion here under a PR called "improve instrumentation" but it really just "removed instrumentation" </span><a href="https://protect2.fireeye.com/v1/url?k=2e829118-7256987e-2e82d183-8610d8a762ca-ad5f363b6140578a&q=1&e=1be7bb71-b40f-43f4-b893-b461dea42001&u=https%3A%2F%2Fgithub.com%2Ferlang%2Fotp%2Fpull%2F1790" target="_blank"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">https://github.com/erlang/otp/pull/1790</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">There are BIF debug functions under hipe that can print this information. hipe_bifs:show_heap/1 and friends.<o:p></o:p></span></p>
</div>
</div>
</div>
<div id="yahoo_quoted_6777507682">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">On Wednesday, April 8, 2020, 02:53:49 p.m. EDT, Dániel Szoboszlay <</span><a href="mailto:dszoboszlay@gmail.com"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">dszoboszlay@gmail.com</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">>
wrote: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<div id="yiv5226392344">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hi,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Even though you cannot trace on allocating large binaries, you may try tracing garbage collections, and look for GC-s that clean up a lot of off-heap binary
data. This could at least narrow done the search for some processes, although it won't tell you where the allocation happens. But maybe once you know which processes are guilty you will be able to add more targeted tracing until you find the root cause.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Cheers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Daniel<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
<div id="yiv5226392344yqt83720">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">On Wed, 8 Apr 2020 at 16:14, Lukas Larsson <</span><a href="mailto:lukas@erlang.org" target="_blank"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">lukas@erlang.org</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">>
wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello,<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">On Tue, Apr 7, 2020 at 9:26 AM Devon Estes <</span><a href="mailto:devon.c.estes@gmail.com" target="_blank"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">devon.c.estes@gmail.com</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">>
wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hi all,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">I’m seeing some cases in my application where our off-process binary heap allocation jumps by several orders of magnitude and then goes down right after. I’m
sure this is something that’s in our app just loading dozens of huge binaries into memory at once and not a bug in anything underlying or a binary leak, but finding where these allocations are happening so I can make some changes to avoid this has so far not
yielded any results. Ideally I’d like to be able to set a trace with something like erlang:trace/3 on some function that sends a tracer message whenever a binary over 30MB is allocated and includes the call stack or even just the calling function that allocated
the binary in the trace message. <o:p></o:p></span></p>
</div>
</blockquote>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Going through the binary vheap and getting a list of the processes that have references to those binaries won’t help in this case.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Is such a trace possible? Is there some flag I can set when starting my BEAM process to give me some kind of debug output that would give me this information?
I’d imagine this is all in C, so it might be a bit tricky...<o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">No it is not possible without modifying the VM. I can't think of any good way to get this information without scanning the process' vheap, which as you say
would not help much in this case.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Thanks in advance for the help!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Cheers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Devon<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">--
<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><br>
_________________<br>
Devon Estes<br>
</span><a href="http://203.559.0323" target="_blank"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">203.559.0323</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><br>
</span><a href="https://protect2.fireeye.com/v1/url?k=719ddf44-2d49d622-719d9fdf-8610d8a762ca-5946e3e7cb52b61a&q=1&e=1be7bb71-b40f-43f4-b893-b461dea42001&u=http%3A%2F%2Fwww.devonestes.com%2F" target="_blank"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">www.devonestes.com</span></a><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>