[erlang-patches] Proposed expansion of Erlang-triggerable DTrace probes

[Scott Lystig Fritchie]

Hi, all.  I've got a proposal for expanding the number of
triggerable-by-Erlang-code DTrace probes.

There's a hassle with the current implementation: it isn't possible to
create truly dynamic DTrace probes(*).  So the method in the R15B01
release creates a single generic probe that Erlang code can fire: you
have 4 integers and 4 strings/binaries/iodata to give to that single
probe.

But there's a problem with the spamminess of the things.  From a DTrace
point of view, there's only a single probe.  So when you enable that one
probe, you automatically get overhead for *all* Erlang-level probe
statements that share that one probe.  You can reduce the overhead by
using pattern-matching-style guards to limit what gets executed,
e.g. arg2 == 703 in order to really fire a probe when arg2 (the first of
four probe-specified integers) is 703 (where 703 == some category of
probe event used by your app).  (See below for an example.)

(*) Except that it _is_ possible, using the technique found in
https://github.com/chrisa/libusdt.  The problem is that we'd like to
have SystemTap users be able to use the same probes as DTrace users, if
at all possible.

So, to solve this problem, we need to:

    1. create more probes that Erlang code can trigger
    2. use libusdt to create truly-dynamic probes

These patches follow method #1, fetch via:

git fetch git://github.com/slfritchie/otp.git slf-dtrace-nif-N-probes

Review via:

https://github.com/slfritchie/otp/compare/erlang:master...slf-dtrace-nif-N-probes
https://github.com/slfritchie/otp/compare/erlang:master...slf-dtrace-nif-N-probes.patch

The patches add 1024 new probes, each with 4 integers and 4 strings.
1024 isn't strictly necessary: the number could be smaller, though I
believe that 256 is probably the smallest useful number.

I could also entertain the notion of adding:

    * 256 probes with 4 integers & 4 strings
    * 256 probes with 8 integers
    * 256 probes with 8 strings
    * N probes with some other combination of integers & strings

Thoughts/opinions?  This isn't a fully-baked submission, quite yet.

-Scott

P.S.  Here's the example that I promised.  Say that I've got an
application that has probes that look like this:

    -define(CATEGORY_FOO, 701).
    -define(CATEGORY_BAR, 702).
    -define(CATEGORY_BAZ, 703).

    %% code path #1
    dyntrace:p(?CATEGORY_FOO, length(SomeList)).
    [...]

    %% code path #2
    dyntrace:p(?CATEGORY_BAZ, size(SomeBinary), "doin' some stuff").

Say that you're interested in the probe in code path #2.  You can do
this via D to print something whenever that probe fires:

    dtrace -qn 'erlang*:::user_trace* /arg2 == 703/ {printf("pid %s: size %d string %s\n", copyinstr(arg0), arg2, copyinstr(arg6));}'

However, if code path #2 is only running 10 times/sec, and code path #1
is running 20K times/sec, the "/arg2 == 703/" guard will help lower the
runtime cost of the 20,010 times/sec by a small amount.  But it would be
much better if we could have a single probe firing only 10 times/sec.