et

v1.8

Event Tracer (ET) Release Notes

This document describes the changes made to the Event Tracer (ET) system from version to version. The intention of this document is to list all incompatibilities as well as all enhancements and bugfixes for every release of Event Tracer (ET). Each release of Event Tracer (ET)thus constitutes one section in this document. The title of each section is the version number of Event Tracer (ET).

ET 1.8

Improvements and New Features

Only minor internal changes.

Own Id: OTP-19964

ET 1.7.3

Improvements and New Features

Release applications, tests, and documentation are now placed in their respective directories. Source SBOM with more packages.

A make release application places only the necessary code in the release folder. The main change is that the documentation and examples are not part of the release folder anymore.

make release_docs places the documentation in the released code under the doc folder.

make release_tests places the tests in their own directory. It used to be the case that some source code was mixed with the tests, and this should not happen anymore.

The Software Bill of Materials places the examples folders as if they are part of the SPDX-otp-<app>-doc packge, instead of placing examples as if they were running source code.

Overall, this change cleans up many things that were not quite correct by definition, and everything should still continue to work as expected. To test a release, one can still run ./Install -minimal \pwd`and add the release to thePATH. After that, one can run tests as usual, going into the released tests directory, enteringtest_server` and running the emulator.

Improves the source Software-Bill-of-Materials

The improvements adds new SPDX relations for asmjit and zlib to be optional_components_of the Erlang/OTP project.

The autoconf scripts in make and erts have now been categorised as build_tool_of the Erlang/OTP project.

All remaining configure, configure.ac, config.h.in, Makefile.in, Makefile.src, EMakefile, and GNUMakefile are now part of a specific SPDX package with relation build_tool_of the Erlang/OTP project.

Own Id: OTP-19886 Aux Id: PR-10434

ET 1.7.2

Improvements and New Features

The license and copyright header has changed format to include an SPDX-License-Identifier. At the same time, most files have been updated to follow a uniform standard for license headers.

Own Id: OTP-19575 Aux Id: PR-9670

ET 1.7.1

Fixed Bugs and Malfunctions

The dependencies for this application are now listed in the app file.

Own Id: OTP-18831 Aux Id: PR-7441

Dialyzer warnings due to type specs added in dbg have been eliminated.

Own Id: OTP-18860

Improvements and New Features

The documentation has been migrated to use Markdown and ExDoc.

Own Id: OTP-18955 Aux Id: PR-8026

ET 1.7

Improvements and New Features

Deprecates dbg:stop_clear/0 because it is simply a function alias to dbg:stop/0

Own Id: OTP-18478 Aux Id: GH-6903

ET 1.6.5

Improvements and New Features

Fixed warnings in code matching on underscore prefixed variables.

Own Id: OTP-17385 Aux Id: OTP-17123

ET 1.6.4

Fixed Bugs and Malfunctions

The scroll bar of the et_viewer window could not be dragged all the way to the top of the window. It would always stop at the second event. This is now corrected.

Own Id: OTP-15463 Aux Id: ERL-780

ET 1.6.3

Fixed Bugs and Malfunctions

Improved documentation.

Own Id: OTP-15190

ET 1.6.2

Improvements and New Features

Calls to erlang:get_stacktrace() are removed.

Own Id: OTP-14861

ET 1.6.1

Improvements and New Features

Tools are updated to show Unicode atoms correctly.

Own Id: OTP-14464

ET 1.6

Improvements and New Features

Update selector to utilize new garbage collection trace tags.

Own Id: OTP-13545

ET 1.5.1

Fixed Bugs and Malfunctions

The et application must continue to use erlang:now/0 in order to obtain timestamps that are consistent with timestamps obtained from tracing. The application has been updated to suppress the warning for erlang:now/0.

Own Id: OTP-12780

ET 1.5

Fixed Bugs and Malfunctions

Application upgrade (appup) files are corrected for the following applications:

asn1, common_test, compiler, crypto, debugger, dialyzer, edoc, eldap, erl_docgen, et, eunit, gs, hipe, inets, observer, odbc, os_mon, otp_mibs, parsetools, percept, public_key, reltool, runtime_tools, ssh, syntax_tools, test_server, tools, typer, webtool, wx, xmerl

A new test utility for testing appup files is added to test_server. This is now used by most applications in OTP.

(Thanks to Tobias Schlager)

Own Id: OTP-11744

Improvements and New Features

Removed gs based applications and gs based backends. The observer application replaces the removed applications.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-10915

ET 1.4.4.5

Improvements and New Features

The encoding of the notes.xml file has been changed from latin1 to utf-8 to avoid future merge problems.

Own Id: OTP-11310

ET 1.4.4.4

Improvements and New Features

Use erlang:demonitor(Ref, [flush]) where applicable. Thanks to Loïc Hoguin.

Own Id: OTP-11039

Rename and document lists:zf/2 as lists:filtermap/2. Thanks to Anthony Ramine.

Own Id: OTP-11078

ET 1.4.4.3

Improvements and New Features

Where necessary a comment stating encoding has been added to Erlang files. The comment is meant to be removed in Erlang/OTP R17B when UTF-8 becomes the default encoding.

Own Id: OTP-10630

ET 1.4.4.2

Fixed Bugs and Malfunctions

Fix typo in ET doc (Thanks to Ricardo Catalinas Jiménez)

Own Id: OTP-10119

ET 1.4.4.1

Improvements and New Features

The GS applications is now deprecated and will be removed in the R16 release. The following GS-based applications have been superseded by the Observer application and will removed in R16: Appmon, Pman, Tv.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-9907

ET 1.4.4

Fixed Bugs and Malfunctions

Dialyzer warnings have been fixed.

Own Id: OTP-9470

ET 1.4.3

Fixed Bugs and Malfunctions

The popup window 'contents viewer' did not display properly on Windows.

Own Id: OTP-9238

ET 1.4.2

Fixed Bugs and Malfunctions

Fix error when module et was used in et_selector trace patterns.

Own Id: OTP-8904

ET 1.4.1

Fixed Bugs and Malfunctions

Fixed broken links in the documentation.

Own Id: OTP-8796

Improvements and New Features

Fixed gui crash on windows.

Own Id: OTP-8830

ET 1.4

Improvements and New Features

Thanks to Olle Mattsson the GUI parts are rewritten to use wxWidgets. For the time being it is still possible to use the old GS based version of the tool, but it is deprecated. The wxWidgets based version is started by default.

A print function has been added, in order to enable printing of sequence charts.

A new tutorial has been added to the documentation. It is based on Jayson Vantuyl's article http://souja.net/2009/04/making-sense-of-erlangs-event-tracer.html.

The functions et:trace_me/4 and et:trace_me/5 has been introduced in order to replace the deprecated functions et:report_event/4 and et:report_event/5. Hopefully the new names makes it a little more obvious what the intended usage of the functions are.

The max_events configuration parameter to et_viewer is not used any more. Now the event cache in the Viewer only contains those events that actually are displayed in the GUI.

Own Id: OTP-8058

ET 1.3.3

Improvements and New Features

The documentation is now built with open source tools (xsltproc and fop) that exists on most platforms. One visible change is that the HTML frames are removed.

Own Id: OTP-8201

ET 1.3.2

Improvements and New Features

The start module in the app file has been corrected.

Own Id: OTP-8078

ET 1.3.1

Improvements and New Features

Changed copyright

Own Id: OTP-7830

ET 1.3

Improvements and New Features

Adjusted copyright

Own Id: OTP-6985

ET 1.0.1

Improvements and New Features

Document source changed from SGML to XML.

Own Id: OTP-6774

ET 1.0.0.1

Improvements and New Features

Minor Makefile changes.

Own Id: OTP-6689

Event Tracer (ET) 1.0

Improvements and new features

This is the first release of the Event Tracer (ET) as a stand-alone application separated from the Megaco application.

Introduction

The Event Tracer (ET) uses the built-in trace mechanism in Erlang and provides tools for collection and graphical viewing of trace data.

The viewed trace data is normally collected from Erlang trace ports or files.

Scope and Purpose

This manual describes the Event Tracer (ET) application, as a component of the Erlang/Open Telecom Platform development environment. It is assumed that the reader is familiar with the Erlang Development Environment, which is described in a separate User's Guide.

Prerequisites

The following prerequisites are required for understanding the material in the Event Tracer (ET) User's Guide:

familiarity with the Erlang system and Erlang programming in general and the especially the art of Erlang tracing.

The application requires Erlang/OTP release R13BB or later. If you use the old GS based GUI it does suffice with R7B.

About This Manual

In addition to this introductory chapter, the Event Tracers User's Guide contains the following chapters:

Chapter 2: "Tutorial" provides a walk-through of the various parts of the application. The tutorial is based on Jayson Vantuyl's article http://souja.net/2009/04/making-sense-of-erlangs-event-tracer.html.

Chapter 3: "Description" describes the architecture and typical usage of the application.

Chapter 4: "Advanced examples" gives some usage examples

Where to Find More Information

Refer to the following documentation for more information about Event Tracer (ET) and about the Erlang/OTP development system:

the Reference Manual of the Event Tracer (ET).

documentation of basic tracing in erlang:trace/3 and erlang:trace_pattern/3 and then the utilities derived from these: dbg, observer, invisio and et.

Programming Erlang: Software for a Concurrent World by Joe Armstrong; ISBN: 978-1-93435-600-5

Tutorial

Visualizing Message Sequence Charts

The easiest way of using ET, is to just use it as a graphical tool for displaying message sequence charts. In order to do that you need to first start a Viewer (which by default starts a Collector):

      {ok, ViewerPid} = et_viewer:start([{title,"Coffee Order"}]),
      CollectorPid = et_viewer:get_collector_pid(ViewerPid).

Then you send events to the Collector with the function et_collector:report_event/6 like this:

      et_collector:report_event(CollectorPid,85,from,to,message,extra_stuff).

The Viewer will automatically pull events from the Collector and display them on the screen.

The number (in this case 85) is an integer from 1 to 100 that specifies the "detail level" of the message. The higher the number, the more important it is. This provides a crude form of priority filtering.

The from, to, and message parameters are exactly what they sound like. from and to are visualized in the Viewer as "lifelines", with the message passing from one to the other. If from and to are the same value, then it is displayed next to the lifeline as an "action". The extra_stuffvalue is simply data that you can attach that will be displayed when someone actually clicks on the action or message in the Viewer window.

The module et/examples/et_display_demo.erl illustrates how it can be used:

-module(et_display_demo).

-export([test/0]).

test() ->
    {ok, Viewer} = et_viewer:start([{title,"Coffee Order"}, {max_actors,10}]),
    Drink = {drink,iced_chai_latte},
    Size = {size,grande},
    Milk = {milk,whole},
    Flavor = {flavor,vanilla},
    C = et_viewer:get_collector_pid(Viewer),
    et_collector:report_event(C,99,customer,barrista1,place_order,[Drink,Size,Milk,Flavor]),
    et_collector:report_event(C,80,barrista1,register,enter_order,[Drink,Size,Flavor]),
    et_collector:report_event(C,80,register,barrista1,give_total,"$5"),
    et_collector:report_event(C,80,barrista1,barrista1,get_cup,[Drink,Size]),
    et_collector:report_event(C,80,barrista1,barrista2,give_cup,[]),
    et_collector:report_event(C,90,barrista1,customer,request_money,"$5"),
    et_collector:report_event(C,90,customer,barrista1,pay_money,"$5"),
    et_collector:report_event(C,80,barrista2,barrista2,get_chai_mix,[]),
    et_collector:report_event(C,80,barrista2,barrista2,add_flavor,[Flavor]),
    et_collector:report_event(C,80,barrista2,barrista2,add_milk,[Milk]),
    et_collector:report_event(C,80,barrista2,barrista2,add_ice,[]),
    et_collector:report_event(C,80,barrista2,barrista2,swirl,[]),
    et_collector:report_event(C,80,barrista2,customer,give_tasty_beverage,[Drink,Size]),
    ok.

When you run the et_display_demo:test(). function in the example above, the Viewer window will look like this:

Four Modules

The event tracer framework is made up of four modules:

et

et_collector

et_viewer

et_selector

In addition, you'll probably want to familiarize yourself with the dbg module and possibly seq_trace module as well.

The Event Tracer Interface

The et module is not like other modules. It contains a function called et:trace_me/5. Which is a function that does not do any useful stuff at all. Its sole purpose is to be a function that is easy to trace. A call to it may be something like:

      et:trace_me(85,from,to,message,extra_stuff).

The parameters to et:trace_me/5 are the same as to et_collector:report_event/6 in the previous chapter. The big difference between the two is in the semantics of the two functions. The second actually reports an Event to the Collector while the first does nothing, it just returns the atom hopefully_traced. In order to make the parameters to et:trace_me/5 turn up in the Collector, tracing of that function must be activated and the Collector must be registered as a Tracer of the Raw Trace Data.

Erlang tracing is a seething pile of pain that involves reasonably complex knowledge of clever ports, tracing return formats, and specialized tracing MatchSpecs (which are really their own special kind of hell). The tracing mechanism is very powerful indeed, but it can be hard to grasp.

Luckily there is a simplified way to start tracing of et:trace_me/5 function calls. The idea is that you should instrument your code with calls to et:trace_me/5 in strategic places where you have interesting information available in your program. Then you just start the Collector with global tracing enabled:

      et_viewer:start([{trace_global, true}, {trace_pattern, {et,max}}]).

This will start a Collector, a Viewer and also start the tracing of et:trace_me/5 function calls. The Raw Trace Data is collected by the Collector and a view of it is displayed on the screen by the Viewer. You can define your own "views" of the data by implementing your own Filter functions and register them in the Viewer.

The Collector and Viewer

These two pieces work in concert. Basically, the Collector receives Raw Trace Data and processes it into Events in a et specific format (defined in et/include/et.hrl). The Viewer interrogates the Collector and displays an interactive representation of the data.

You might wonder why these aren't just one module. The Collector is a generic full-fledged framework that allows processes to "subscribe" to the Events that it collects. One Collector can serve several Viewers. The typical case is that you have one Viewer that visualizes Events in one flavor and another Viewer that visualizes them in another flavor. If you for example are tracing a text based protocol like HTML (or Megaco/H.248) it would be useful to be able to display the Events as plain text as well as the internal representation of the message. The architecture does also allow you to implement your own Viewer program as long as it complies to the protocol between the Collector/Viewer protocol. Currently two kinds of Viewers exists. That is the old GS based one and the new based on wxWidgets. But if you feel for it you may implement your own Viewer, which for example could display the Events as ASCII art or whatever you feel useful.

The Viewer will by default create a Collector for you. With a few options and some configuration settings you can start collecting Events.

The Collector API does also allow you to save the collected Events to file and later load them in a later session.

The Selector

This is perhaps the most central module in the entirety of the et suite. The Collector needs "filters" to convert the Raw Trace Data into "events" that it can display. The et_selector module provides the default Filter and some API calls to manage the Trace Pattern. The Selector provides various functions that achieve the following:

Convert Raw Trace Data into an appropriate Event

Magically notice traces of the et:trace_me/5 function and make appropriate Events

Carefully prevent translating the Raw Trace Data twice

Manage a Trace Pattern

The Trace Pattern is basically a tuple of a module and a detail level (either an integer or the atom max for full detail). In most cases the Trace Pattern {et,max} does suffice. But if you do not want any runtime dependency of et you can implement your own trace_me/5 function in some module and refer to that module in the Trace Pattern.

The specified module flows from your instantiation of the Viewer, to the Collector that it automatically creates, gets stashed in as the Trace Pattern, and eventually goes down into the bowels of the Selector.

The module that you specify gets passed down (eventually) into Selector's default Filter. The format of the et:trace_me/5 function call is hardcoded in that Filter.

How To Put It Together

The Collector automatically registers itself to listen for trace Events, so all you have to do is enable them.

For those people who want to do general tracing, consult the dbg module on how to trace whatever you're interested in and let it work its magic. If you just want et:trace_me/5 to work, do the following:

Create a Collector

Create a Viewer (this can do step #1 for you)

Turn on and pare down debugging

The module et/examples/et_trace_demo.erl achieves this.

-module(et_trace_demo).

-export([test/0]).

test() ->
    et_viewer:start([
        {title,"Coffee Order"},
        {trace_global,true},
        {trace_pattern,{et,max}},
        {max_actors,10}
      ]),
      %% dbg:p(all,call),
      %% dbg:tpl(et, trace_me, 5, []),
      Drink = {drink,iced_chai_latte},
      Size = {size,grande},
      Milk = {milk,whole},
      Flavor = {flavor,vanilla},
      et:trace_me(99,customer,barrista1,place_order,[Drink,Size,Milk,Flavor]),
      et:trace_me(80,barrista1,register,enter_order,[Drink,Size,Flavor]),
      et:trace_me(80,register,barrista1,give_total,"$5"),
      et:trace_me(80,barrista1,barrista1,get_cup,[Drink,Size]),
      et:trace_me(80,barrista1,barrista2,give_cup,[]),
      et:trace_me(90,barrista1,customer,request_money,"$5"),
      et:trace_me(90,customer,barrista1,pay_money,"$5"),
      et:trace_me(80,barrista2,barrista2,get_chai_mix,[]),
      et:trace_me(80,barrista2,barrista2,add_flavor,[Flavor]),
      et:trace_me(80,barrista2,barrista2,add_milk,[Milk]),
      et:trace_me(80,barrista2,barrista2,add_ice,[]),
      et:trace_me(80,barrista2,barrista2,swirl,[]),
      et:trace_me(80,barrista2,customer,give_tasty_beverage,[Drink,Size]),
      ok.

Running through the above, the most important points are:

Turn on global tracing

Set a Trace Pattern

Tell dbg to trace function Calls

Tell it specifically to trace the et:trace_me/5 function

When you run the et_trace_demo:test() function above, the Viewer window will look like this screenshot:

Description

Overview

The two major components of the Event Tracer (ET) tool is a graphical sequence chart viewer (et_viewer) and its backing storage (et_collector). One Collector may be used as backing storage for several simultaneous Viewers where each one may display a different view of the same trace data.

The interface between the Collector and its Viewers is public in order to enable other types of Viewers. However in the following text we will focus on usage of the et_viewer.

The main start function is et_viewer:start/1. By default it will start both an et_collector and an et_viewer:

% erl -pa et/examples
Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> {ok, Viewer} = et_viewer:start([]).
{ok,<0.40.0>}

A Viewer gets trace Events from its Collector by polling it regularly for more Events to display. Events are for example reported to the Collector with et_collector:report_event/6:

2> Collector = et_viewer:get_collector_pid(Viewer).
<0.39.0>
3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, start_outer,
3>                           "Start outer transaction"),
3> et_collector:report_event(Collector, 40, mnesia_tm, my_shell, new_tid,
3>                           "New transaction id is 4711"),
3> et_collector:report_event(Collector, 20, my_shell, mnesia_locker, try_write_lock,
3>                           "Acquire write lock for {my_tab, key}"),
3> et_collector:report_event(Collector, 10, mnesia_locker, my_shell, granted,
3>                           "You got the write lock for {my_tab, key}"),
3> et_collector:report_event(Collector, 60, my_shell, do_commit,
3>                           "Perform  transaction commit"),
3> et_collector:report_event(Collector, 40, my_shell, mnesia_locker, release_tid,
3>                           "Release all locks for transaction 4711"),
3> et_collector:report_event(Collector, 60, my_shell, mnesia_tm, delete_transaction,
3>                           "End of outer transaction"),
3> et_collector:report_event(Collector, 20, my_shell, end_outer,
3>                           "Transaction returned {atomic, ok}").
{ok,{table_handle,<0.39.0>,16402,trace_ts,
     #Fun<et_collector.0.62831470>}}

This actually is a simulation of the process Events caused by a Mnesia transaction that writes a record in a local table:

mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).

At this stage when we have a couple of Events, it is time to show how it looks like in the graphical interface of et_viewer:

In the sequence chart, the actors (which symbolically has performed the Event) are shown as named vertical bars. The order of the actors may be altered by dragging (hold mouse button 1 pressed during the operation) the name tag of an actor and drop it elsewhere:

An Event may be an action performed by one single actor (blue text label) or it may involve two actors and is then depicted as an arrow directed from one actor to another (red text label). Details of an Event can be shown by clicking (press and release the mouse button 1) on the event label text or on the arrow. When doing that a Contents Viewer window pops up. It may look like this:

Filters and dictionary

The Event Tracer (ET) uses named filters in various contexts. An Event Trace filter is an Erlang fun that takes some trace data as input and returns a possibly modified version of it:

filter(TraceData) -> false | true | {true, NewEvent}

TraceData = Event | erlang_trace_data()
Event = #event{}
NewEvent = #event{}

The interface of the filter function is the same as the the filter functions for the good old lists:filtermap/2. If the filter returns false it means that the trace data should silently be dropped. true means that the trace data data already is an Event Record and that it should be kept as it is. true means that the TraceData already is an Event Record and that it should be kept as it is. {true, NewEvent} means that the original trace data should be replaced with Event. This provides means to get rid of unwanted Events as well as enabling alternate views of an Event.

The first filter that the trace data is exposed for is the Collector Filter. When a trace Event is reported with et_collector:report/2 (or et_collector:report_event/5,6) the first thing that happens, is that a message is sent to the Collector process to fetch a handle that contains some useful stuff, such as the Collector Filter Fun and an Ets table identifier. Then the Collector Filter Fun is applied and if it returns true (or {true, NewEvent}), the Event will be stored in an Ets table. As an optimization, subsequent calls to et_collector:report-functions can use the handle directly instead of the Collector Pid.

All filters (registered in a Collector or in a Viewer) must be able to handle an Event record as input. The Collector Filter (that is the filter named all) is a little bit special, as its input also may be raw Erlang Trace Data

The Collector manages a key/value based dictionary, where the filters are stored. Updates of the dictionary is propagated to all subscribing processes. When a Viewer is started it is registered as a subscriber of dictionary updates.

In each Viewer there is only one filter that is active and all trace Events that the Viewer gets from the Collector will pass thru that filter. By writing clever filters it is possible to customize how the Events looks like in the viewer. The following filter in et/examples/et_demo.erl replaces the actor names mnesia_tm and mnesia_locker and leaves everything else in the record as it was:

mgr_actors(E) when is_record(E, event) ->
    Actor = fun(A) ->
               case A of
                   mnesia_tm     -> trans_mgr;
                   mnesia_locker -> lock_mgr;
                   _             -> A
               end
            end,
    {true, E#event{from = Actor(E#event.from),
                   to = Actor(E#event.to),
                   contents = [{orig_from, E#event.from},
                               {orig_to,   E#event.to},
                               {orig_contents, E#event.contents}]}}.

If we now add the filter to the running Collector:

4> Fun = fun(E) -> et_demo:mgr_actors(E) end.
#Fun<erl_eval.6.13229925>
5> et_collector:dict_insert(Collector, {filter, mgr_actors}, Fun).
ok

you will see that the Filter menu in all viewers have got a new entry called mgr_actors. Select it, and a new Viewer window will pop up:

In order to see the nitty gritty details of an Event you may click on the Event in order to start a Contents Viewer for that Event. In the Contents Viewer there also is a filter menu that enables inspection of the Event from other views than the one selected in the viewer. A click on the new_tid Event will cause a Contents Viewer window to pop up, showing the Event in the mgr_actors view:

Select the all entry in the Filters menu and a new Contents Viewer window will pop up showing the same trace Event in the collectors view:

Trace clients

As you have seen, it is possible to use the et_collector:report_event/5,6 functions explicitly. By using those functions you can write your own trace client that reads trace data from any source stored in any format and just feed the Collector with it. You may replace the default Collector Filter with a filter that converts new exciting trace data formats to Event Records or you may convert it to an Event Record before you invoke et_collector:report/2 and then rely on the default Collector Filter to handle the new format.

There are also existing functions in the API that reads from various sources and calls et_collector:report/2:

The trace Events that are hosted by the Collector may be stored to file and later be loaded by selecting save and load entries in the Viewers File menu or via the et_collector API.

It is also possible to perform live tracing of a running system by making use of the built-in trace support in the Erlang emulator. These Erlang traces can be directed to files or to ports. See the reference manual for erlang:trace/3, erlang:trace_pattern/3, dbg and ttb for more info.

There are also corresponding trace client types that can read the Erlang trace data format from such files or ports. The et_collector:start_trace_client/3 function makes use of these Erlang trace clients and redirects the trace data to the Collector.

The default Collector Filter converts the raw Erlang trace data format into Event Records. If you want to perform this differently you can of course write your own Collector Filter from scratch. But it may probably save you some efforts if you first apply the default filter in et_selector:parse_event/2 before you apply your own conversions of its output.

Global tracing

Setting up an Erlang tracer on a set of nodes and connecting trace clients to the ports of these tracers is not intuitive. In order to make this it easier the Event Tracer has a notion of global tracing. When used, the et_collector process will monitor Erlang nodes and when one connects, an Erlang tracer will automatically be started on the newly connected node. A corresponding trace client will also be started on the Collector node in order to automatically forward the trace Events to the Collector. Set the boolean parameter trace_global to true for either the et_collector or et_viewer in order to activate the global tracing. There is no restriction on how many concurrent (anonymous) collectors you can have, but you can only have one global Collector as its name is registered in global.

In order to further simplify the tracing, you can make use of the et:trace_me/4,5 functions. These functions are intended to be invoked from other applications when there are interesting Events, in your application that needs to be highlighted. The functions are extremely light weight as they do nothing besides returning an atom. These functions are specifically designed to be traced for. As the caller explicitly provides the values for the Event Record fields, the default Collector Filter is able to automatically provide a customized Event Record without any user defined filter functions.

In normal operation, the et:trace_me/4,5 calls are almost for free. When tracing is needed, you can either activate tracing on these functions explicitly. Or you can combine the usage of trace_global with the usage of trace_pattern. When set, the trace_pattern will automatically be activated on all connected nodes.

One nice thing with the trace_pattern is that it provides a very simple way of minimizing the amount of generated trace data by allowing you to explicitly control the detail level of the tracing. As you may have seen the et_viewer have a slider called "Detail Level" that allows you to control the detail level of the trace Events displayed in the Viewer. On the other hand if you set a low detail level in the trace_pattern, lots of the trace data will never be generated and thus not sent over the socket to the trace client and stored in the Collector.

Viewer window

Almost all functionality available in the et_viewer is also available via shortcuts. Which key that has the same effect as selecting a menu entry is shown enclosed in parentheses. For example pressing the key r is equivalent to selecting the menu entry Viewer->Refresh.

File menu:

Clear all events in the Collector - Deletes all Events stored in the Collector and notifies all connected Viewers about this.

Load events to the Collector from file - Loads the Collector with Events from a file and notifies all connected Viewers about this.

Save all events in the Collector to file - Saves all Events stored in the Collector to file.

Print setup - Enables editing of printer setting, such as paper and layout.

Print current page - Prints the events on the current page. The page size is dependent of the selected paper type.

Print all pages - Prints all events. The page size is dependent of the selected paper type.

Close this Viewer - Closes this Viewer window, but keeps all other Viewers windows and the Collector process.

Close other Viewers, but this - Keeps this Viewer window and its Collector process, but closes all other Viewers windowsconnected to the same Collector.

Close all Viewers and the Collector - Closes the Collector and all Viewers connected to it.

Viewer menu:

First - Scrolls this viewer to the first Event in the Collector.

Last - Scrolls this viewer to the last Event in the Collector.

Prev - Scrolls this viewer one page backwards.

Next - Scrolls this viewer one page forward.

Refresh - Clears this viewer and re-read its Events from the Collector.

Up - Scrolls a few Events backwards.

Down - Scrolls a few Events forward.

Display all actors. - Reset the settings for hidden and/or highlighted actors.

Collector menu:

First - Scrollsall viewers to the first Event in the Collector.

Last - Scrolls all viewers to the last Event in the Collector.

Prev - Scrolls all viewers one page backwards.

Next - Scrolls all viewers one page forward.

Refresh - Clears all viewers and re-read their Events from the Collector.

Filters and scaling menu:

ActiveFilter (=) - Starts a new Viewer window with the same active filter and scale as the current one.

ActiveFilter (+) - Starts a new Viewer window with the same active filter but a larger scale than the current one.

ActiveFilter (-) - Starts a new Viewerwindow with the same active filter but a smaller scale than the current one.

all (0) - Starts a new Viewer with the Collector Filter as active filter. It will cause all events in the collector to be viewed.

AnotherFilter (2) - If more filters are inserted into the dictionary, these will turn up here as entries in the Filters menu. The second filter will get the shortcut number 2, the next one number 3 etc. The names are sorted.

Slider and radio buttons:

Hide From=To - When true, this means that the Viewer will hide all Events where the from-actor equals to its to-actor. These events are sometimes called actions.

Hide (excluded actors) - When true, this means that the Viewer will hide all Events whose actors are marked as excluded. Excluded actors are normally enclosed in round brackets when they are displayed in the Viewer.

Detail level - This slider controls the resolution of the Viewer. Only Events with a detail level smaller than the selected one (default=100=max) are displayed.

Other features:

Vertical scroll - Use mouse wheel and up/down arrows to scroll little. Use page up/down and home/end buttons to scroll more.

Display details of an event - Left mouse click on the event label or the arrowand a new Contents Viewer window will pop up, displaying the contents of an Event.

Highlight actor (toggle) - Left mouse click on the actor name tag. The actor name will be enclosed in square brackets []. When one or more actors are highlighted, only events related to those actors are displayed. All others are hidden.

Exclude actor (toggle) - Right mouse click on the actor name tag. The actor name will be enclosed in round brackets (). When an actor is excluded, all events related to this actor is hidden. If the checkbox Hide (excluded actors) is checked, even the name tags and corresponding vertical line of excluded actors will be hidden.

Move actor - Left mouse button drag and drop on actor name tag. Move the actor by first clicking on the actor name, keeping the button pressed while moving the cursor to a new location and release the button where the actor should be moved to.

Display all actors - Press the 'a' button. Reset the settings for hidden and/or highlighted actors.

Configuration

The Event Records in the Ets table are ordered by their timestamp. Which timestamp that should be used is controlled via the event_order parameter. Default is trace_ts which means the time when the trace data was generated. event_ts means the time when the trace data was parsed (transformed into an Event Record).

Contents viewer window

File menu:

Close - Close this window.

Save - Save the contents of this window to file.

Filters menu:

ActiveFilter - Start a new Contents Viewer window with the same active filter.

AnotherFilter (2) - If more filters are inserted into the dictionary, these will turn up here as entries in the Filters menu. The second filter will be number 2, the next one number 3 etc. The names are sorted.

Hide menu:

Hide actor in viewer - Known actors are shown as a named vertical bars in the Viewer window. By hiding the actor, its vertical bar will be removed and the Viewer will be refreshed.

Hiding the actor is only useful if the max_actors threshold has been reached, as it then will imply that the "hidden" actor will be displayed as if it were "UNKNOWN". If the max_actors threshold not have been reached, the actor will re-appear as a vertical bar in the Viewer.

Show actor in viewer - This implies that the actor will be added as a known actor in the Viewer with its own vertical bar.

Search menu:

Forward from this event - Set this event to be the first event in the viewer and change its display mode to be enter forward search mode. The actor of this event (from, to or both) will be added to the list of selected actors.

Reverse from this event - Set this event to be the first Event in the Viewer and change its display mode to be enter reverse search mode. The actor of this Event (from, to or both) will be added to the list of selected actors. Observe, that the Events will be shown in reverse order.

Abort search. Display all - Switch the display mode of the Viewer to show all Events regardless of any ongoing searches. Abort the searches.

Advanced examples

A simulated Mnesia transaction

The Erlang code for running the simulated Mnesia transaction example in the previous chapter is included in the et/examples/et_demo.erl file:

sim_trans() ->
    sim_trans([]).

sim_trans(ExtraOptions) ->
    Options = [{dict_insert, {filter, mgr_actors}, fun mgr_actors/1}],
    {ok, Viewer} = et_viewer:start_link(Options ++ ExtraOptions),
    Collector = et_viewer:get_collector_pid(Viewer),
    et_collector:report_event(Collector, 60, my_shell, mnesia_tm, start_outer,
                              "Start outer transaction"),
    et_collector:report_event(Collector, 40, mnesia_tm, my_shell, new_tid,
                              "New transaction id is 4711"),
    et_collector:report_event(Collector, 20, my_shell, mnesia_locker, try_write_lock,
                              "Acquire write lock for {my_tab, key}"),
    et_collector:report_event(Collector, 10, mnesia_locker, my_shell, granted,
                              "You got the write lock for {my_tab, key}"),
    et_collector:report_event(Collector, 60, my_shell, do_commit,
                              "Perform  transaction commit"),
    et_collector:report_event(Collector, 40, my_shell, mnesia_locker, release_tid,
                              "Release all locks for transaction 4711"),
    et_collector:report_event(Collector, 60, my_shell, mnesia_tm, delete_transaction,
                              "End of outer transaction"),
    et_collector:report_event(Collector, 20, my_shell, end_outer,
                              "Transaction returned {atomic, ok}"),
    {collector, Collector}.

mgr_actors(E) when is_record(E, event) ->
    Actor = fun(A) ->
               case A of
                   mnesia_tm     -> trans_mgr;
                   mnesia_locker -> lock_mgr;
                   _             -> A
               end
            end,
    {true, E#event{from = Actor(E#event.from),
                   to = Actor(E#event.to),
                   contents = [{orig_from, E#event.from},
                               {orig_to,   E#event.to},
                               {orig_contents, E#event.contents}]}}.

If you invoke the et_demo:sim_trans() function, a Viewer window will pop up and the sequence trace will be almost the same as if the following Mnesia transaction would have been run:

mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).

And the viewer window will look like:

Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> {ok, Viewer} = et_viewer:start([]).
{ok,<0.40.0>;}
2> et_demo:sim_trans().
{ok,{table_handle,<0.45.0>,24596,trace_ts,
     #Fun<et_collector.0.62831470>}}

Some convenient functions used in the Mnesia transaction example

The module_as_actor filter converts the Event Records so the module names becomes actors and the invoked functions becomes labels. If the information about who the caller was it will be displayed as an arrow directed from the caller to the callee. The [{message, {caller}}, {return_trace}] options to dbg:tpl/2 function will imply the necessary information in the Erlang traces. Here follows the module_as_actor filter:

module_as_actor(E) when is_record(E, event) ->
    case lists:keysearch(mfa, 1, E#event.contents) of
        {value, {mfa, {M, F, _A}}} ->
            case lists:keysearch(pam_result, 1, E#event.contents) of
                {value, {pam_result, {M2, _F2, _A2}}} ->
                    {true, E#event{label = F, from = M2, to = M}};
                _ ->
                    {true, E#event{label = F, from = M, to = M}}
            end;
        _ ->
            false
    end.

The plain_process_info filter does not alter the Event Records. It merely ensures that the event not related to processes are skipped:

plain_process_info(E) when is_record(E, event) ->
    case E#event.label of
        send                          -> true;
        send_to_non_existing_process  -> true;
        'receive'                     -> true;
        spawn                         -> true;
        exit                          -> true;
        link                          -> true;
        unlink                        -> true;
        getting_linked                -> true;
        {seq_send, _Label}            -> true;
        {seq_receive, _Label}         -> true;
        {seq_print, _Label}           -> true;
        {drop, _N}                    -> true;
        _                             -> false
    end.

The plain_process_info_nolink filter does not alter the Event Records. It do makes use of the plain_process_info , but do also ensure that the process info related to linking and unlinking is skipped:

plain_process_info_nolink(E) when is_record(E, event) ->
    (E#event.label /= link) and
    (E#event.label /= unlink) and
    (E#event.label /= getting_linked) and
    plain_process_info(E).

In order to simplify the startup of an et_viewer process with the filters mentioned above, plus some others (that also are found in et/examples/et_demo.erl src/et_collector.erl the et_demo:start/0,1 functions can be used:

start() ->
    start([]).

start(ExtraOptions) ->
    Options = [{trace_global, true},
               {parent_pid, undefined},
               {max_actors, infinity},
               {max_events, 1000},
               {active_filter, module_as_actor}],
    et_viewer:start_link(filters() ++ Options ++ ExtraOptions).

A simple one-liner starts the tool:

          erl -pa ../examples -s et_demo

The filters are included by the following parameters:

filters() ->
    [{dict_insert, {filter, module_as_actor},
                   fun module_as_actor/1},
     {dict_insert, {filter, plain_process_info},
                   fun plain_process_info/1},
     {dict_insert, {filter, plain_process_info_nolink},
                   fun plain_process_info_nolink/1},
     {dict_insert, {filter, named_process_info},
                   fun named_process_info/1},
     {dict_insert, {filter, named_process_info_nolink},
                   fun named_process_info_nolink/1},
     {dict_insert, {filter, node_process_info},
                   fun node_process_info/1},
     {dict_insert, {filter, node_process_info_nolink},
                   fun node_process_info_nolink/1},
     {dict_insert, {filter, application_as_actor},
                   fun application_as_actor/1}
    ].

Erlang trace of a real Mnesia transaction

The following piece of code et_demo:trace_mnesia/0 activates call tracing of both local and external function calls for all modules in the Mnesia application. The call traces are configured cover all processes (both existing and those that are spawned in the future) and include timestamps for trace data. It do also activate tracing of process related events for Mnesia's static processes plus the calling process (that is your shell). Please, observe that the whereis/1 call in the following code requires that both the traced Mnesia application and the et_viewer is running on the same node:

trace_mnesia() ->
    Modules = mnesia:ms(),
    Spec = [{message, {caller}}, {return_trace}],
    Flags = [send, 'receive', procs, timestamp],
    dbg:p(all, [call, timestamp]),
    [dbg:tpl(M, [{'_', [], Spec}]) || M <- Modules],
    LocallyRunningServers = [M || M <- Modules, whereis(M) /= undefined],
    [dbg:p(whereis(RS), Flags) || RS <- LocallyRunningServers],
    dbg:p(self(), Flags),
    LocallyRunningServers.

The et_demo:live_trans/0 function starts the global Collector, starts a Viewer, starts Mnesia, creates a local table, activates tracing (as described above) and registers the shell process is as 'my_shell' for clarity. Finally a simple Mnesia transaction that writes a single record is run:

live_trans() ->
    live_trans([]).

live_trans(ExtraOptions) ->
    Options = [{title, "Mnesia tracer"},
	       {hide_actions, true},
	       {active_filter, named_process_info_nolink}],
    et_demo:start(Options ++ ExtraOptions),
    mnesia:start(),
    mnesia:create_table(my_tab, [{ram_copies, [node()]}]),
    et_demo:trace_mnesia(),
    register(my_shell, self()),

    mnesia:transaction(fun() -> mnesia:write({my_tab, key, val}) end).

Now we run the et_demo:live_trans/0 function:

erl -pa ../examples
Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4]
                           [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
1> et_demo:live_trans().
{atomic,ok}

Please, explore the different filters in order to see how the traced transaction can be seen from different point of views:

Erlang trace of Megaco startup

The Event Tracer (ET) tool was initially written in order to demonstrate how messages where sent over the Megaco protocol. This were back in the old days before the standard bodies of IETF and ITU had approved Megaco (also called H.248) as an international standard.

In the Megaco application of Erlang/OTP, the code is carefully instrumented with calls to et:trace_me/5. For each call a detail level is given in order to enable dynamic control of the trace level in a simple manner.

The megaco_filter module implements a customized filter for Megaco messages. It does also make use of trace_global combined with usage of the trace_pattern:

-module(megaco_filter).
-export([start/0]).

start() ->
    Options =
        [{event_order, event_ts},
         {scale, 3},
         {max_actors, infinity},
         {trace_pattern, {megaco, max}},
         {trace_global, true},
         {dict_insert, {filter, megaco_filter}, fun filter/1},
         {active_filter, megaco_filter},
         {title, "Megaco tracer - Erlang/OTP"}],
    et_viewer:start(Options).

First we start an Erlang node with a global Collector and its Viewer.

erl -sname observer
Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
(observer@falco)1> megaco_filter:start().
{ok,<0.48.0>}

Secondly we start another Erlang node which we connect the observer node, before we start the application that we want to trace. In this case we start a Media Gateway Controller that listens for both TCP and UDP on the text and binary ports for Megaco:

erl -sname mgc -pa ../../megaco/examples/simple
Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
(mgc@falco)1> net:ping(observer@falco).
pong
(mgc@falco)2> megaco:start().
ok
(mgc@falco)3> megaco_simple_mgc:start().
{ok,[{ok,2944,
         {megaco_receive_handle,{deviceName,"controller"},
                                megaco_pretty_text_encoder,[],megaco_tcp,dynamic}},
     {ok,2944,
         {megaco_receive_handle,{deviceName,"controller"},
                                megaco_pretty_text_encoder,[],megaco_udp,dynamic}},
     {ok,2945,
         {megaco_receive_handle,{deviceName,"controller"},
                                megaco_binary_encoder,[],megaco_tcp,dynamic}},
     {ok,2945,
         {megaco_receive_handle,{deviceName,"controller"},
                                megaco_binary_encoder,[],megaco_udp,dynamic}}]}

And finally we start an Erlang node for the Media Gateways and connect to the observer node. Each Media Gateway connects to the controller and sends an initial Service Change message. The controller accepts the gateways and sends a reply to each one using the same transport mechanism and message encoding according to the preference of each gateway. That is all combinations of TCP/IP transport, UDP/IP transport, text encoding and ASN.1 BER encoding:

Erlang R13B03 (erts-5.7.4) [64-bit] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.4  (abort with ^G)
(mg@falco)1> net:ping(observer@falco).
pong
(mg@falco)2> megaco_simple_mg:start().
[{{deviceName,"gateway_tt"},
  {error,{start_user,megaco_not_started}}},
 {{deviceName,"gateway_tb"},
  {error,{start_user,megaco_not_started}}},
 {{deviceName,"gateway_ut"},
  {error,{start_user,megaco_not_started}}},
 {{deviceName,"gateway_ub"},
  {error,{start_user,megaco_not_started}}}]
(mg@falco)3> megaco:start().
ok
(mg@falco)4> megaco_simple_mg:start().
[{{deviceName,"gateway_tt"},
  {1,
   {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
            [{serviceChangeReply,
                 {'ServiceChangeReply',
                     [{megaco_term_id,false,["root"]}],
                     {serviceChangeResParms,
                         {'ServiceChangeResParm',
                             {deviceName,"controller"},
                             asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE,
                             asn1_NOVALUE}}}}]}]}}},
 {{deviceName,"gateway_tb"},
  {1,
   {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
            [{serviceChangeReply,
                 {'ServiceChangeReply',
                     [{megaco_term_id,false,["root"]}],
                     {serviceChangeResParms,
                         {'ServiceChangeResParm',
                             {deviceName,"controller"},
                             asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE,
                             asn1_NOVALUE}}}}]}]}}},
 {{deviceName,"gateway_ut"},
  {1,
   {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
            [{serviceChangeReply,
                 {'ServiceChangeReply',
                     [{megaco_term_id,false,["root"]}],
                     {serviceChangeResParms,
                         {'ServiceChangeResParm',
                             {deviceName,"controller"},
                             asn1_NOVALUE,asn1_NOVALUE,asn1_NOVALUE,
                             asn1_NOVALUE}}}}]}]}}},
 {{deviceName,"gateway_ub"},
  {1,
   {ok,[{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
            [{serviceChangeReply,
                 {'ServiceChangeReply',
                     [{megaco_term_id,false,["root"]}],
                     {serviceChangeResParms,
                         {'ServiceChangeResParm',
                             {deviceName,"controller"},
                             asn1_NOVALUE,asn1_NOVALUE,
                             asn1_NOVALUE,...}}}}]}]}}}]

The Megaco adopted viewer looks like this, when we have clicked on the [gateway_tt] actor name in order to only display the events regarding that actor:

A pretty printed Megaco message looks like this:

And the corresponding internal form for the same Megaco message looks like this:

et

Main API of the Event Trace (ET) application

Interface module for the Event Trace (ET) application

Summary

Types

actor()

level()

Functions

phone_home(DetailLevel, FromTo, Label, Contents)

Equivalent to phone_home/5.

phone_home(DetailLevel, From, To, Label, Contents)

These functions sends a signal to the outer space and the caller hopes that someone is listening. In other words, they invoke et:trace_me/4 and et:trace_me/5 respectively.

report_event(DetailLevel, FromTo, Label, Contents)

Equivalent to report_event/5.

report_event(DetailLevel, From, To, Label, Contents)

Deprecated functions which for the time being are kept for backwards compatibility. Invokes et:trace_me/4 and et:trace_me/5 respectively.

trace_me(DetailLevel, FromTo, Label, Contents)

Invokes et:trace_me/5 with both From and To set to FromTo.

trace_me(DetailLevel, From, To, Label, Contents)

A function that is intended to be traced.

Types

actor()

(not exported)

-type actor() :: term().

level()

(not exported)

-type level() :: 0..100.

Functions

phone_home(DetailLevel, FromTo, Label, Contents)

-spec phone_home(DetailLevel, FromTo, Label, Contents) -> hopefully_traced
                    when
                        DetailLevel :: level(),
                        FromTo :: actor(),
                        Label :: atom() | string() | term(),
                        Contents :: [{Key :: term(), Value :: term()}] | term().

Equivalent to phone_home/5.

phone_home(DetailLevel, From, To, Label, Contents)

-spec phone_home(DetailLevel, From, To, Label, Contents) -> hopefully_traced
                    when
                        DetailLevel :: level(),
                        From :: actor(),
                        To :: actor(),
                        Label :: atom() | string() | term(),
                        Contents :: [{Key :: term(), Value :: term()}] | term().

These functions sends a signal to the outer space and the caller hopes that someone is listening. In other words, they invoke et:trace_me/4 and et:trace_me/5 respectively.

report_event(DetailLevel, FromTo, Label, Contents)

-spec report_event(DetailLevel, FromTo, Label, Contents) -> hopefully_traced
                      when
                          DetailLevel :: level(),
                          FromTo :: actor(),
                          Label :: atom() | string() | term(),
                          Contents :: [{Key :: term(), Value :: term()}] | term().

Equivalent to report_event/5.

report_event(DetailLevel, From, To, Label, Contents)

-spec report_event(DetailLevel, From, To, Label, Contents) -> hopefully_traced
                      when
                          DetailLevel :: level(),
                          From :: actor(),
                          To :: actor(),
                          Label :: atom() | string() | term(),
                          Contents :: [{Key :: term(), Value :: term()}] | term().

Deprecated functions which for the time being are kept for backwards compatibility. Invokes et:trace_me/4 and et:trace_me/5 respectively.

trace_me(DetailLevel, FromTo, Label, Contents)

(since OTP R13B04)

-spec trace_me(DetailLevel, FromTo, Label, Contents) -> hopefully_traced
                  when
                      DetailLevel :: level(),
                      FromTo :: actor(),
                      Label :: atom() | string() | term(),
                      Contents :: [{Key :: term(), Value :: term()}] | term().

Invokes et:trace_me/5 with both From and To set to FromTo.

trace_me(DetailLevel, From, To, Label, Contents)

(since OTP R13B04)

-spec trace_me(DetailLevel, From, To, Label, Contents) -> hopefully_traced
                  when
                      DetailLevel :: level(),
                      From :: actor(),
                      To :: actor(),
                      Label :: atom() | string() | term(),
                      Contents :: [{Key :: term(), Value :: term()}] | term().

A function that is intended to be traced.

This function is intended to be invoked at strategic places in user applications in order to enable simplified tracing. The functions are extremely light weight as they do nothing besides returning an atom. The functions are designed for being traced. The global tracing mechanism in et_collector defaults to set its trace pattern to these functions.

The label is intended to provide a brief summary of the event. It is preferred to use an atom but a string would also do.

The contents can be any term but in order to simplify post processing of the traced events, a plain list of {Key, Value} tuples is preferred.

Some events, such as messages, are directed from some actor to another. Other events (termed actions) may be undirected and only have one actor.

et_collector

Collect trace events and provide a backing storage appropriate for iteration

Interface module for the Event Trace (ET) application

Summary

Types

actor()

collector_fun()

dbg_match_spec()

dbg_trace_parameters()

dbg_trace_type()

ets_match_object_pattern()

event()

event_filter_fun()

event_ts()

level()

option()

table_handle()

trace_filter_fun()

trace_ts()

Functions

change_pattern(CollectorPid, RawPattern)

Change active trace pattern globally on all trace nodes.

clear_table(Handle)

Clear the event table.

dict_delete(CollectorPid, Key)

Delete a dictionary entry and send a {et, {dict_delete, Key}} tuple to all registered subscribers.

dict_insert(CollectorPid, What, Value)

Insert a dictionary entry and send a {et, {dict_insert, Key, Val}} tuple to all registered subscribers.

dict_lookup(CollectorPid, Key)

Lookup a dictionary entry and return zero or one value.

dict_match(CollectorPid, Pattern)

Match some dictionary entries

get_global_pid()

Return a the identity of the globally registered collector if there is any.

iterate(Handle, Prev, Limit)

Equivalent to iterate(Handle, Prev, Limit, undefined, Prev).

iterate(Handle, Prev, Limit, Fun, Acc)

Iterate over the currently stored events.

make_key(Handle, Stuff)

Make a key out of an event record or an old key.

multicast(CollectorPid, Msg)

Sends a message to all registered subscribers.

report(Handle, TraceOrEvent)

Equivalent to report_event/6.

report_event(Handle, DetailLevel, FromTo, Label, Contents)

Equivalent to report_event/6.

report_event(Handle, DetailLevel, From, To, Label, Contents)

Report an event to the collector.

save_event_file(CollectorPid, FileName, Options)

Save the events to a file.

start_link(Options)

Start a collector process.

start_trace_client(CollectorPid, Type, Parameter)

Load raw Erlang trace from a file, port or process.

stop(CollectorPid)

Stop a collector process.

Types

actor()

(not exported)

-type actor() :: term().

collector_fun()

(not exported)

-type collector_fun() :: trace_filter_fun() | event_filter_fun().

dbg_match_spec()

(not exported)

-type dbg_match_spec() :: [tuple()].

dbg_trace_parameters()

(not exported)

-type dbg_trace_parameters() :: file:filename() | integer() | {string(), integer()}.

dbg_trace_type()

(not exported)

-type dbg_trace_type() :: ip | file | follow_file.

ets_match_object_pattern()

(not exported)

-type ets_match_object_pattern() :: term().

event()

(not exported)

-type event() ::
          #event{detail_level :: term(),
                 trace_ts :: term(),
                 event_ts :: term(),
                 from :: term(),
                 to :: term(),
                 label :: term(),
                 contents :: term()}.

event_filter_fun()

(not exported)

-type event_filter_fun() :: fun((Event :: event()) -> boolean() | {true, event()}).

event_ts()

(not exported)

-type event_ts() :: #event_ts{event_ts :: term(), trace_ts :: term()}.

level()

(not exported)

-type level() :: 0..100.

option()

-type option() ::
          {parent_pid, pid() | undefined} |
          {event_order, trace_ts | event_ts} |
          {dict_insert, {filter, all}, collector_fun()} |
          {dict_insert, {filter, EventFilterName :: atom()}, event_filter_fun()} |
          {dict_insert, {subscriber, pid()}, Val :: term()} |
          {dict_insert, Key :: term(), Val :: term()} |
          {dict_delete, Key :: term()} |
          {trace_client, {event_file, file:filename()} | {dbg_trace_type(), dbg_trace_parameters()}} |
          {trace_global, boolean()} |
          {trace_pattern, {module() | undefined, Level :: level() | dbg_match_spec()} | undefined} |
          {trace_port, integer()} |
          {trace_max_queue, integer()}.

table_handle()

(not exported)

-type table_handle() ::
          #table_handle{collector_pid :: term(),
                        event_tab :: term(),
                        event_order :: term(),
                        filter :: term()}.

trace_filter_fun()

(not exported)

-type trace_filter_fun() :: fun((TraceData :: tuple()) -> boolean() | {true, event()}).

trace_ts()

(not exported)

-type trace_ts() :: #trace_ts{trace_ts :: term(), event_ts :: term()}.

Functions

change_pattern(CollectorPid, RawPattern)

-spec change_pattern(CollectorPid, RawPattern) -> {old_pattern, TracePattern}
                        when
                            CollectorPid :: pid(),
                            RawPattern :: {module(), min | max | level()},
                            TracePattern :: [{[term()] | '_' | atom(), [term()], [term()]}].

Change active trace pattern globally on all trace nodes.

clear_table(Handle)

-spec clear_table(Handle) -> ok when Handle :: CollectorPid | table_handle(), CollectorPid :: pid().

Clear the event table.

dict_delete(CollectorPid, Key)

-spec dict_delete(CollectorPid :: pid(), Key :: term()) -> ok.

Delete a dictionary entry and send a {et, {dict_delete, Key}} tuple to all registered subscribers.

If the deleted entry is a registered subscriber, it will imply that the subscriber process gets is unregistered as subscriber as well as it gets it final message.

dict_insert(CollectorPid, What, Value)

-spec dict_insert(CollectorPid, What, Value) -> ok
                     when
                         CollectorPid :: pid(),
                         What :: {filter, atom()} | {subscriber, pid()} | Key,
                         Key :: term(),
                         Value :: collector_fun() | term().

Insert a dictionary entry and send a {et, {dict_insert, Key, Val}} tuple to all registered subscribers.

If the entry is a new subscriber, it will imply that the new subscriber process first will get one message for each already stored dictionary entry, before it and all old subscribers will get this particular entry. The collector process links to and then supervises the subscriber process. If the subscriber process dies it will imply that it gets unregistered as with a normal dict_delete/2.

dict_lookup(CollectorPid, Key)

-spec dict_lookup(CollectorPid :: pid(), Key :: term()) -> [Val :: term()].

Lookup a dictionary entry and return zero or one value.

dict_match(CollectorPid, Pattern)

-spec dict_match(CollectorPid :: pid(), {KeyPattern, ValPattern}) -> [Match]
                    when
                        KeyPattern :: ets_match_object_pattern(),
                        ValPattern :: ets_match_object_pattern(),
                        Match :: {Key :: term(), Val :: term()}.

Match some dictionary entries

get_global_pid()

-spec get_global_pid() -> CollectorPid :: pid().

Return a the identity of the globally registered collector if there is any.

iterate(Handle, Prev, Limit)

-spec iterate(Handle, Prev, Limit) -> NewAcc
                 when
                     Handle :: CollectorPid | table_handle(),
                     CollectorPid :: pid(),
                     Prev :: first | last | term(),
                     Limit :: Done | Forward | Backward,
                     Done :: 0,
                     Forward :: pos_integer(),
                     Backward :: neg_integer(),
                     NewAcc :: term().

Equivalent to iterate(Handle, Prev, Limit, undefined, Prev).

iterate(Handle, Prev, Limit, Fun, Acc)

-spec iterate(Handle, Prev, Limit, Fun, Acc) -> NewAcc
                 when
                     Handle :: CollectorPid | table_handle(),
                     CollectorPid :: pid(),
                     Prev :: first | last | term(),
                     Limit :: Done | Forward | Backward,
                     Done :: 0,
                     Forward :: pos_integer() | infinity,
                     Backward :: neg_integer() | '-infinity',
                     Fun :: fun((event(), Acc) -> NewAcc) | undefined,
                     Acc :: term(),
                     NewAcc :: term().

Iterate over the currently stored events.

Iterates over the currently stored events and applies a function for each event. The iteration may be performed forwards or backwards and may be limited to a maximum number of events (abs(Limit)).

make_key(Handle, Stuff)

-spec make_key(Handle, Stuff) -> Key
                  when
                      Handle :: table_handle() | trace_ts | event_ts,
                      Stuff :: event() | Key,
                      Key :: event_ts() | trace_ts().

Make a key out of an event record or an old key.

multicast(CollectorPid, Msg)

-spec multicast(CollectorPid :: pid(), Msg :: term()) -> ok.

Sends a message to all registered subscribers.

report(Handle, TraceOrEvent)

-spec report(Handle, TraceOrEvent) -> {ok, Continuation}
                when
                    Handle :: table_handle() | (CollectorPid :: pid()),
                    TraceOrEvent :: event() | TraceData | end_of_trace,
                    TraceData :: tuple(),
                    Continuation :: table_handle().

Equivalent to report_event/6.

report_event(Handle, DetailLevel, FromTo, Label, Contents)

-spec report_event(Handle, DetailLevel, FromTo, Label, Contents) -> {ok, Continuation}
                      when
                          Handle :: CollectorPid :: pid() | table_handle(),
                          DetailLevel :: level(),
                          FromTo :: actor(),
                          Label :: term(),
                          Contents :: [{Key :: term(), Value :: term()}] | term(),
                          Continuation :: table_handle().

Equivalent to report_event/6.

report_event(Handle, DetailLevel, From, To, Label, Contents)

-spec report_event(Handle, DetailLevel, From, To, Label, Contents) -> {ok, Continuation}
                      when
                          Handle :: CollectorPid :: pid() | table_handle(),
                          DetailLevel :: level(),
                          From :: actor(),
                          To :: actor(),
                          Label :: term(),
                          Contents :: [{Key :: term(), Value :: term()}] | term(),
                          Continuation :: table_handle().

Report an event to the collector.

All events are filtered thru the collector filter, which optionally may transform or discard the event. The first call should use the pid of the collector process as report handle, while subsequent calls should use the table handle.

save_event_file(CollectorPid, FileName, Options)

-spec save_event_file(CollectorPid, FileName, [Option]) -> ok | {error, term()}
                         when
                             CollectorPid :: pid(),
                             FileName :: file:filename(),
                             Option :: existing | write | append | keep | clear.

Save the events to a file.

By default the currently stored events (existing) are written to a brand new file (write) and the events are kept (keep) after they have been written to the file.

Instead of keeping the events after writing them to file, it is possible to remove all stored events after they have successfully written to file (clear).

The options defaults to existing, write and keep.

start_link(Options)

-spec start_link(Options) -> {ok, Pid :: pid()} | {error, term()} when Options :: [option()].

Start a collector process.

The collector collects trace events and keeps them ordered by their timestamp. The timestamp may either reflect the time when the actual trace data was generated (trace_ts) or when the trace data was transformed into an event record (event_ts). If the time stamp is missing in the trace data (missing timestamp option to erlang:trace/4) the trace_ts will be set to the event_ts.

Events are reported to the collector directly with the report function or indirectly via one or more trace clients. All reported events are first filtered thru the collector filter before they are stored by the collector. By replacing the default collector filter with a customized dito it is possible to allow any trace data as input. The collector filter is a dictionary entry with the predefined key {filter, collector} and the value is a fun of arity 1. See et_selector:make_event/1 for interface details, such as which erlang:trace/1 tuples that are accepted.

The collector has a built-in dictionary service. Any term may be stored as value in the dictionary and bound to a unique key. When new values are inserted with an existing key, the new values will overwrite the existing ones. Processes may subscribe on dictionary updates by using {subscriber, pid()} as dictionary key. All dictionary updates will be propagated to the subscriber processes matching the pattern {{subscriber, ''}, ''} where the first '_' is interpreted as a pid().

In global trace mode, the collector will automatically start tracing on all connected Erlang nodes. When a node connects, a port tracer will be started on that node and a corresponding trace client on the collector node.

Default values:

parent_pid - self().

event_order - trace_ts.

trace_global - false.

trace_pattern - undefined.

trace_port - 4711.

trace_max_queue - 50.

start_trace_client(CollectorPid, Type, Parameter)

-spec start_trace_client(CollectorPid, Type, Parameter) -> file_loaded | {trace_client_pid, pid()}
                            when
                                CollectorPid :: pid(),
                                Type :: event_file | dbg_trace_type(),
                                Parameter :: dbg_trace_parameters().

Load raw Erlang trace from a file, port or process.

stop(CollectorPid)

-spec stop(pid()) -> ok.

Stop a collector process.

et_selector

Define event transforms and trace patterns

Summary

Types

event()

level()

Functions

change_pattern(Pattern)

Activates/deactivates tracing by changing the current trace pattern.

make_pattern(RawPattern)

Makes a trace pattern suitable to feed change_pattern/1

parse_event(Mod, ValidTraceData)

Transforms trace data and makes an event record out of it.

Types

event()

(not exported)

-type event() ::
          #event{detail_level :: term(),
                 trace_ts :: term(),
                 event_ts :: term(),
                 from :: term(),
                 to :: term(),
                 label :: term(),
                 contents :: term()}.

level()

(not exported)

-type level() :: 0..100.

Functions

change_pattern(Pattern)

-spec change_pattern({Mod :: module(), Pattern}) -> ok
                        when
                            Pattern :: DetailLevel | TracePattern | EmptyTracePattern,
                            DetailLevel :: level(),
                            TracePattern :: [{[term()] | '_' | atom(), [term()], [term()]}],
                            EmptyTracePattern :: [].

Activates/deactivates tracing by changing the current trace pattern.

min detail level deactivates tracing of calls to et:trace_me/4,5

max detail level activates tracing of all calls to et:trace_me/4,5

integer(X) detail level activates tracing of all calls to et:trace_me/4,5 whose detail level argument is lesser than X.

An empty match spec deactivates tracing of calls to et:trace_me/4,5

Other match specs activates tracing of calls to et:trace_me/4,5 accordingly with erlang:trace_pattern/2.

make_pattern(RawPattern)

-spec make_pattern({Mod :: module(), RawPattern}) -> {Mod :: module(), TracePattern}
                      when
                          RawPattern :: level(),
                          TracePattern :: [{[term()] | '_' | atom(), [term()], [term()]}].

Makes a trace pattern suitable to feed change_pattern/1

Min detail level deactivates tracing of calls to et:trace_me/4,5

Max detail level activates tracing of all calls to et:trace_me/4,5

integer(X) detail level activates tracing of all calls to et:trace_me/4,5 whose detail level argument is lesser than X.

See also erlang:trace_pattern/2 for more info about its match_spec()

parse_event(Mod, ValidTraceData)

-spec parse_event(Mod, ValidTraceData) -> boolean | {true, event()}
                     when
                         Mod :: module(),
                         ValidTraceData :: ErlangTraceData | event(),
                         ErlangTraceData ::
                             {trace, pid(), atom(), term()} |
                             {trace, pid(), atom(), term(), term()} |
                             {trace_ts, pid(), atom(), term(), TS :: {integer(), integer(), integer()}} |
                             {trace_ts,
                              pid(),
                              atom(),
                              term(),
                              term(),
                              TS :: {integer(), integer(), integer()}} |
                             {seq_trace, atom(), term()} |
                             {seq_trace, atom(), term(), TS :: {integer(), integer(), integer()}} |
                             {drop, integer()}.

Transforms trace data and makes an event record out of it.

See erlang:trace/3 for more info about the semantics of the trace data.

An event record consists of the following fields:

detail_level - Noise has a high level as opposed to essentials.

trace_ts - Time when the trace was generated. Same as event_ts if omitted in trace data.

event_ts - Time when the event record was created.

from - From actor, such as sender of a message.

to - To actor, such as receiver of message.

label - Label intended to provide a brief event summary.

contents - All nitty gritty details of the event.

See et:trace_me/4and et:trace_me/5 for details.

Returns:

{true, Event} - where Event is an #event{} record representing the trace data

true - means that the trace data already is an event record and that it is valid as it is. No transformation is needed.

false - means that the trace data is uninteresting and should be dropped

et_viewer

Displays a sequence chart for trace events (messages/actions)

Summary

Types

actors()

first_key()

option()

Functions

file(FileName)

Start a new event viewer and a corresponding collector and load them with trace events from a trace file.

get_collector_pid(ViewerPid)

Returns the identifier of the collector process.

start()

Simplified start of a sequence chart viewer with global tracing activated.

start(GUIorOptions)

Start of a sequence chart viewer without linking to the parent process.

start_link(GUIorOptions)

Start a sequence chart viewer for trace events (messages/actions)

stop(ViewerPid)

Stops a viewer process.

Types

actors()

(not exported)

-type actors() :: [term()].

first_key()

(not exported)

-type first_key() :: term().

option()

(not exported)

-type option() ::
          {title, string()} |
          {detail_level, 0..100} |
          {is_suspended, boolean()} |
          {scale, integer()} |
          {width, integer()} |
          {height, integer()} |
          {collector_pid, pid() | undefined} |
          {active_filter, atom()} |
          {max_events, integer() | undefined} |
          {max_actors, integer() | undefined} |
          {actors, actors()} |
          {first_event, first_key()} |
          {hide_unknown, boolean()} |
          {hide_actions, boolean()} |
          {display_mode, all | {search_actors, forward | reverse, first_key(), actors()}}.

Functions

file(FileName)

-spec file(FileName :: file:filename()) -> {ok, pid()} | {error, term()}.

Start a new event viewer and a corresponding collector and load them with trace events from a trace file.

get_collector_pid(ViewerPid)

-spec get_collector_pid(ViewerPid :: pid()) -> pid().

Returns the identifier of the collector process.

start()

-spec start() -> {ok, pid()} | {error, term()}.

Simplified start of a sequence chart viewer with global tracing activated.

Convenient to be used from the command line (erl -s et_viewer).

start(GUIorOptions)

-spec start(GUIorOptions) -> {ok, Viewer :: pid()} | {error, term()}
               when
                   GUIorOptions :: wx | default | Options, Options :: [option() | et_collector:option()].

Start of a sequence chart viewer without linking to the parent process.

start_link(GUIorOptions)

-spec start_link(GUIorOptions) -> {ok, Viewer :: pid()} | {error, term()}
                    when
                        GUIorOptions :: wx | default | Options,
                        Options :: [option() | et_collector:option()].

Start a sequence chart viewer for trace events (messages/actions)

A filter_fun() takes an event record as sole argument and returns false | true | {true, NewEvent}.

If the collector_pid is undefined a new et_collector will be started with the following parameter settings: parent_pid, event_order, trace_global, trace_pattern, trace_port, trace_max_queue, trace_client, dict_insert and dict_delete. The new et_viewer will register itself as an et_collector subscriber.

Default values:

parent_pid - self().

title - "et_viewer".

detail_level - max.

is_suspended - false.

scale - 2.

width - 800.

height - 600.

collector_pid - undefined.

event_order - trace_ts.

active_filter - collector.

max_actors - 5.

actors - ["UNKNOWN"].

first_event - first.

hide_unknown - false.

hide_actions - false.

display_mode - all.

stop(ViewerPid)

-spec stop(ViewerPid :: pid()) -> ok.

Stops a viewer process.