Obtain message content of Erlang messages in dtrace

[Harris, Robert]

> On 16 Dec 2019, at 12:02, Duncan Paul Attard <duncan.attard.01@REDACTED> wrote:
>
> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.

TL;DR you can use DTrace but you'll have to modify the Erlang VM.

DTrace was intended to provide observation at the very lowest level.  In
the user land case, a process is viewed as a collection of load objects
whose symbol tables describe functions and global variables.  DTrace is
organised as a framework of "providers" that expose certain events ---
locations in executable code or moments in time --- and its "pid"
provider permits the instrumentation of functions at their entry and
return and at any offset between.  One may inspect a process's
environment in terms of its registers and its address space;  also
exposed are integer arguments on function entry and a return value on
exit, but these are simply special cases of register use.

This abstraction is particularly well suited to simple C applications:
tracing function arguments is trivial and often sufficient in its own
right.  With some work to find their locations, local variables are
visible.  If some quantity is a pointer then dereferencing it is
reasonably straight-forward.  Composite data types are fully supported
and may be navigated with varying degrees of ease, depending on the
DTrace implementation.

For more complex programs, the pid provider alone falls short and your
question provides a perfect example.  It may happen that some
significant event, e.g. the receipt of a message, does not correspond to
a well known location such as the entry to or the return from a
function.  Instead, it might lie somewhere within a function at an
unstable offset determined only at compilation.  How may one describe
the location of this event?  Worse, DTrace will reveal only the contents
of a register or a memory location.  Suppose that we obtain the value of
the variable used in the Erlang VM to identify a message;  perhaps it is
a pointer.  To what does it point?  Likely something unwieldy to the
developer and opaque to the user.

DTrace provides a solution in the form of a static tracing point, in
essence a C macro with a stable name that is visible to DTrace.  Thus
the developer can associate a named probe with some logical event (e.g.
message receipt) and expose useful diagnostic data (e.g. a textual
representation of the message itself).

Turning at last to your actual problem, the message-receive probe is
used at line 242 here:

https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/msg_instrs.tab#L227

(Note that variables exposed by the probe are initialised only if the
probe itself is enabled, which is determined by the earlier macro at
227.)

Interestingly, the message itself isn't exposed, which is why you have
been unable to trace it.  However, I imagine it would be feasible to
patch the VM to do what you want.  You would need to render the message
as a string into a buffer and expose that buffer's address;  an existing
example of this idiom is

https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/global.h#L1603

The name for such static DTrace probes is USDT;  you can read more about
it at

https://docs.oracle.com/cd/E37838_01/html/E61035/gkycs.html#scrolltoc

but there may be differences on your OS (e.g. see DTrace(1) on macos).

Robert
Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.