Tracing large binary allocations

[Max Lapshin]

Wow!  Cool thing!

On Mon, Apr 13, 2020 at 3:34 PM Jonas Falkevik <
jonas.falkevik@REDACTED> wrote:

> Hi,
> I was playing around with bpftrace during the weekend.
> And even if it isn’t super great. It can trace some allocations at least.
> Tested on x86_64 Linux 5.5.11
>
> $ ./run_bpftrace
>     usage: ./run_bpftrace <path to beam> <filter size> [summerize]
> $ ./run_bpftrace /home/jonas/src/otp/bin/x86_64-unknown-linux-gnu/beam.smp
> 200000
> Attaching 10 probes...
> erts_alcu_alloc_thr_pref pid <0.19.0> size 347899
> erts_alcu_alloc_thr_pref pid <0.51.0> size 347899
> erts_alcu_alloc_thr_pref pid <0.75.0> size 262175
> erts_alcu_alloc_thr_pref pid <0.75.0> size 262175
> erts_alcu_alloc_thr_pref pid <0.10.0> size 565523
> erts_alcu_alloc_thr_pref pid <0.10.0> size 314640
> erts_alcu_alloc_thr_spec pid <0.10.0> size 210672
> erts_alcu_alloc_thr_spec pid <0.10.0> size 210672
> erts_alcu_alloc_thr_spec pid <0.10.0> size 210672
> erts_alcu_alloc_thr_pref pid <0.10.0> size 302233599
> erts_alcu_alloc_thr_pref pid <0.10.0> size 302233599
> erts_alcu_alloc_thr_pref pid <0.60.0> size 302233599
>
>
> https://github.com/falkevik/erl_bpftrace
>
>
> -Jonas
>
> > On 10 Apr 2020, at 10:14, John Högberg <john.hogberg@REDACTED>
> wrote:
> >
> > Hi,
> >
> > The old instrumentation module couldn’t trace allocations, and your
> method for finding (long-lived) binary leaks required a debugger and manual
> interpretation of block contents which goes far beyond what we can support.
> Still, there’s nothing preventing you from doing that with the new
> instrumentation and I provided brief instructions on how to create the same
> kind of memory map using `gdb` in the thread you linked. I can’t see why
> you insist on saying it’s impossible, it’s merely different.
> >
> > In either case, the problem at hand is transient so going through the
> blocks wouldn’t help and OP said as much in their first post. We’ll look
> into adding tracing of some sort for large allocations, be it `LTTng`
> probes or otherwise.
> >
> > /John
> >
> > From: erlang-questions <erlang-questions-bounces@REDACTED> On Behalf
> Of Vans S
> > Sent: Thursday, April 09, 2020 03:03
> > To: Erlang (E-mail) <erlang-questions@REDACTED>; Dániel Szoboszlay <
> dszoboszlay@REDACTED>
> > Subject: Re: Tracing large binary allocations
> >
> > This used to be possible via the instrument module but the functionality
> got removed in OTP21, I used this functionality to detect a large binheap
> leak in jiffy way back when. Theres a lengthy discussion here under a PR
> called "improve instrumentation" but it really just "removed
> instrumentation" https://github.com/erlang/otp/pull/1790.
> >
> > There are BIF debug functions under hipe that can print this
> information.  hipe_bifs:show_heap/1 and friends.
> > On Wednesday, April 8, 2020, 02:53:49 p.m. EDT, Dániel Szoboszlay <
> dszoboszlay@REDACTED> wrote:
> >
> >
> > Hi,
> >
> > Even though you cannot trace on allocating large binaries, you may try
> tracing garbage collections, and look for GC-s that clean up a lot of
> off-heap binary data. This could at least narrow done the search for some
> processes, although it won't tell you where the allocation happens. But
> maybe once you know which processes are guilty you will be able to add more
> targeted tracing until you find the root cause.
> >
> > Cheers,
> > Daniel
> >
> > On Wed, 8 Apr 2020 at 16:14, Lukas Larsson <lukas@REDACTED> wrote:
> > Hello,
> >
> > On Tue, Apr 7, 2020 at 9:26 AM Devon Estes <devon.c.estes@REDACTED>
> wrote:
> > Hi all,
> >
> > I’m seeing some cases in my application where our off-process binary
> heap allocation jumps by several orders of magnitude and then goes down
> right after. I’m sure this is something that’s in our app just loading
> dozens of huge binaries into memory at once and not a bug in anything
> underlying or a binary leak, but finding where these allocations are
> happening so I can make some changes to avoid this has so far not yielded
> any results. Ideally I’d like to be able to set a trace with something like
> erlang:trace/3 on some function that sends a tracer message whenever a
> binary over 30MB is allocated and includes the call stack or even just the
> calling function that allocated the binary in the trace message.
> >
> > Going through the binary vheap and getting a list of the processes that
> have references to those binaries won’t help in this case.
> >
> > Is such a trace possible? Is there some flag I can set when starting my
> BEAM process to give me some kind of debug output that would give me this
> information? I’d imagine this is all in C, so it might be a bit tricky...
> >
> > No it is not possible without modifying the VM. I can't think of any
> good way to get this information without scanning the process' vheap, which
> as you say would not help much in this case.
> >
> >
> > Thanks in advance for the help!
> >
> > Cheers,
> > Devon
> > --
> >
> > _________________
> > Devon Estes
> > 203.559.0323
> > www.devonestes.com
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200414/049d0f49/attachment.htm>