Obtain message content of Erlang messages in dtrace

[Duncan Paul Attard]

Thanks a lot for your detailed answer!


> On 05 Jan 2020, at 01:39, Harris, Robert <robert.harris@REDACTED> wrote:
> 
> 
> 
>> On 16 Dec 2019, at 12:02, Duncan Paul Attard <duncan.attard.01@REDACTED> wrote:
>> 
>> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.
> 
> TL;DR you can use DTrace but you'll have to modify the Erlang VM.
> 
> DTrace was intended to provide observation at the very lowest level.  In
> the user land case, a process is viewed as a collection of load objects
> whose symbol tables describe functions and global variables.  DTrace is
> organised as a framework of "providers" that expose certain events ---
> locations in executable code or moments in time --- and its "pid"
> provider permits the instrumentation of functions at their entry and
> return and at any offset between.  One may inspect a process's
> environment in terms of its registers and its address space;  also
> exposed are integer arguments on function entry and a return value on
> exit, but these are simply special cases of register use.
> 
> This abstraction is particularly well suited to simple C applications:
> tracing function arguments is trivial and often sufficient in its own
> right.  With some work to find their locations, local variables are
> visible.  If some quantity is a pointer then dereferencing it is
> reasonably straight-forward.  Composite data types are fully supported
> and may be navigated with varying degrees of ease, depending on the
> DTrace implementation.
> 
> For more complex programs, the pid provider alone falls short and your
> question provides a perfect example.  It may happen that some
> significant event, e.g. the receipt of a message, does not correspond to
> a well known location such as the entry to or the return from a
> function.  Instead, it might lie somewhere within a function at an
> unstable offset determined only at compilation.  How may one describe
> the location of this event?  Worse, DTrace will reveal only the contents
> of a register or a memory location.  Suppose that we obtain the value of
> the variable used in the Erlang VM to identify a message;  perhaps it is
> a pointer.  To what does it point?  Likely something unwieldy to the
> developer and opaque to the user.
> 
> DTrace provides a solution in the form of a static tracing point, in
> essence a C macro with a stable name that is visible to DTrace.  Thus
> the developer can associate a named probe with some logical event (e.g.
> message receipt) and expose useful diagnostic data (e.g. a textual
> representation of the message itself).
> 
> Turning at last to your actual problem, the message-receive probe is
> used at line 242 here:
> 
> https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/msg_instrs.tab#L227
> 
> (Note that variables exposed by the probe are initialised only if the
> probe itself is enabled, which is determined by the earlier macro at
> 227.)
> 
> Interestingly, the message itself isn't exposed, which is why you have
> been unable to trace it.  However, I imagine it would be feasible to
> patch the VM to do what you want.  You would need to render the message
> as a string into a buffer and expose that buffer's address;  an existing
> example of this idiom is
> 
> https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/global.h#L1603
> 
> The name for such static DTrace probes is USDT;  you can read more about
> it at
> 
> https://docs.oracle.com/cd/E37838_01/html/E61035/gkycs.html#scrolltoc
> 
> but there may be differences on your OS (e.g. see DTrace(1) on macos).
> 
> Robert
> Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.