runtime_tools

v2.3

Runtime tools Application

Description

This chapter describes the Runtime_Tools application in OTP, which provides low footprint tracing/debugging tools suitable for inclusion in a production system.

Configuration

There are currently no configuration parameters available for this application.

SEE ALSO

application(3)

Runtime_Tools Release Notes

This document describes the changes made to the Runtime_Tools application.

Runtime_Tools 2.3

Fixed Bugs and Malfunctions

NIFs and linked-in drivers are now loadable when running in an Erlang source tree on Windows.

Own Id: OTP-19686 Aux Id: PR-9969

Improvements and New Features

The default tracer is now aware that it is started by a remote shell (-remsh), in which case the traces will be sent to the remote group_leader to make the traces visible in the remote shell.

Own Id: OTP-19648 Aux Id: PR-9589

A User's Guide to dbg is now available in the documentation.

Own Id: OTP-19655 Aux Id: PR-9853

Runtime_Tools 2.2

Improvements and New Features

EEP-69: Nominal Types has been implemented. As a side effect, nominal types can encode opaque types. We changed all opaque-handling logic and improved opaque warnings in Dialyzer.

All existing Erlang type systems are structural: two types are seen as equivalent if their structures are the same. Type comparisons are based on the structures of the types, not on how the user explicitly defines them. For example, in the following example, meter() and foot() are equivalent. The two types can be used interchangeably. Neither of them differ from the basic type integer().

-type meter() :: integer().
-type foot() :: integer().

Nominal typing is an alternative type system, where two types are equivalent if and only if they are declared with the same type name. The EEP proposes one new syntax -nominal for declaring nominal types. Under nominal typing, meter() and foot() are no longer compatible. Whenever a function expects type meter(), passing in type foot() would result in a Dialyzer error.

-nominal meter() :: integer().
-nominal foot() :: integer().

More nominal type-checking rules can be found in the EEP. It is worth noting that most work for adding nominal types and type-checking is in erl_types.erl. The rest are changes that removed the previous opaque type-checking, and added an improved version of it using nominal type-checking with reworked warnings.

Backwards compatibility for opaque type-checking is not preserved by this PR. Previous opaque warnings can appear with slightly different wordings. A new kind of opaque warning opaque_union is added, together with a Dialyzer option no_opaque_union to turn this kind of warnings off.

Own Id: OTP-19364 Aux Id: PR-9079

When compiling C/C++ code on Unix systems, the compiler hardening flags suggested by the Open Source Security Foundation are now enabled by default. To disable them, pass --disable-security-hardening-flags to configure.

Own Id: OTP-19519 Aux Id: PR-9441

The license and copyright header has changed format to include an SPDX-License-Identifier. At the same time, most files have been updated to follow a uniform standard for license headers.

Own Id: OTP-19575 Aux Id: PR-9670

With this change observer will use cheaper iterators to avoid locking when not necessary.

Own Id: OTP-19584 Aux Id: PR-9711

Runtime_Tools 2.1.1

Fixed Bugs and Malfunctions

Fixed a bug where dbg sessions on remote nodes were terminated prematurely.

Own Id: OTP-19188 Aux Id: PR-8692

Runtime_Tools 2.1

Improvements and New Features

The instrument module can now track allocations on a per-process or per-port basis.

Own Id: OTP-18577 Aux Id: PR-7236

The new function proc_lib:set_label/1 can be used to add a descriptive term to any process that does not have a registered name. The name will be shown by tools such as c:i/0, observer, and it will be included in crash reports produced by processes using gen_server, gen_statem, gen_event, and gen_fsm.

The label for a process can be retrieved by calling proc_lib:get_label/1.

Note that those functions work on any process, not only processes that use proc_lib.

Example:

1> self().
<0.90.0>
2> proc_lib:set_label(my_label).
ok
3> i().
    .
    .
    .
<0.90.0>              erlang:apply/2                        2586    75011    0
my_label              c:pinfo/2                               51
4> proc_lib:get_label(self()).
my_label

Own Id: OTP-18789 Aux Id: PR-7720, PR-8003

Type specs had been added to all dbg functions.

Own Id: OTP-18859 Aux Id: PR-7782

The documentation has been migrated to use Markdown and ExDoc.

Own Id: OTP-18955 Aux Id: PR-8026

dbg have been updated to use trace sessions.

dbg:session_create/1, dbg:session/2, and dbg:session_destroy/1 have been added to work sessions. See the documentation for details.

Own Id: OTP-19081 Aux Id: PR-8363

Runtime_Tools 2.0.1

Fixed Bugs and Malfunctions

Fixed issue with fetching port information for observer could crash if port had died.

Own Id: OTP-18868 Aux Id: GH-7735

Fixed some benign compile warnings on Windows.

Own Id: OTP-18895

Runtime_Tools 2.0

Fixed Bugs and Malfunctions

Fixed the type specification for instrument:carriers/0,1

Own Id: OTP-18499 Aux Id: PR-6946

Improvements and New Features

Add dbg:tracer(file, Filename) as a convenient way to trace to a file in clean text.

Own Id: OTP-18211 Aux Id: PR-6143

Handling of on_load modules during boot has been improved by adding an extra step in the boot order for embedded mode that runs all on_load handlers, instead of relying on explicit invocation of them, later, when the kernel supervision tree starts.

This is mostly a code improvement and OTP internal simplification to avoid future bugs and to simplify code maintenance.

Own Id: OTP-18447

Deprecates dbg:stop_clear/0 because it is simply a function alias to dbg:stop/0

Own Id: OTP-18478 Aux Id: GH-6903

The instrument module has been moved from tools to runtime_tools.

Own Id: OTP-18487 Aux Id: PR-6829

Removed the experimental erts_alloc_config module. It no longer produced good configurations and cannot be fixed in a reasonably backwards compatible manner. It has since OTP 25 been deprecated and scheduled for removal in OTP 26.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-18549 Aux Id: PR-7105

Runtime_Tools 1.19

Fixed Bugs and Malfunctions

Reading port socket options on macOS and Windows "skips" invalid options.

Own Id: OTP-18012 Aux Id: #5798

Improvements and New Features

dbg:stop/0 now behaves like dbg:stop_clear/0, clearing all global trace patterns for all functions.

Own Id: OTP-17909 Aux Id: ERIERL-760

erts_alloc_config has been scheduled for removal in OTP 26. It has not produced good configurations for a very long time, and unfortunately it cannot be fixed in a backwards compatible manner.

Own Id: OTP-17939

Runtime_Tools 1.18

Fixed Bugs and Malfunctions

Fixed bug in scheduler:utilization(Seconds) that would leave the scheduler_wall_time system flag incorrectly enabled.

Own Id: OTP-17800 Aux Id: PR-5425

Improvements and New Features

Add scheduler:get_sample/0 and get_sample_all/0. Also clarify scheduler module documentation about how it depends on system flag scheduler_wall_time.

Own Id: OTP-17830 Aux Id: GH-5425, PR-5444

Runtime_Tools 1.17

Improvements and New Features

Observer now has a sectiion for new socket.

Own Id: OTP-17346

The dbg docs have been expanded to include the meaning of all the function name acronyms.

Own Id: OTP-17572 Aux Id: PR-5117

Runtime_Tools 1.16.2

Improvements and New Features

The experimental HiPE application has been removed, together with all related functionality in other applications.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-16963

Runtime_Tools 1.16.1

Fixed Bugs and Malfunctions

The function dbg:n/1 used a local fun to set up a tracer on a remote node. This works fine as long as the remote node is running exactly the same version of Erlang/OTP but does not work at all otherwise. This is fixed by exporting the relevant function and by calling this function on the remote node to set up remote tracing.

Own Id: OTP-16930 Aux Id: ERL-1371, GH-4396

Runtime_Tools 1.16

Improvements and New Features

Clarify documentation of module 'scheduler'.

Own Id: OTP-17208 Aux Id: GH-4502, PR-4532

Runtime_Tools 1.15.1

Fixed Bugs and Malfunctions

Fixed a crash in appmon_info triggered by trying to read port info from a port that was in the process of terminating.

appmon_info is used by observer to get information from the observed node.

Own Id: OTP-16787 Aux Id: PR-2673

Runtime_Tools 1.15

Improvements and New Features

Improved the presentation of allocations and carriers in the instrument module.

Own Id: OTP-16327

Minor updates due to the new spawn improvements made.

Own Id: OTP-16368 Aux Id: OTP-15251

Runtime_Tools 1.14.0.1

Fixed Bugs and Malfunctions

The function dbg:n/1 used a local fun to set up a tracer on a remote node. This works fine as long as the remote node is running exactly the same version of Erlang/OTP but does not work at all otherwise. This is fixed by exporting the relevant function and by calling this function on the remote node to set up remote tracing.

Own Id: OTP-16930 Aux Id: ERL-1371, GH-4396

Runtime_Tools 1.14

Improvements and New Features

Fix dbg:stop_clear/0 to also clear trace events (send and 'receive').

Own Id: OTP-16044

Runtime_Tools 1.13.3

Improvements and New Features

Minor updates to build system necessary due to move of configuration of crypto application.

Own Id: OTP-15262 Aux Id: OTP-15129

Runtime_Tools 1.13.2.1

Fixed Bugs and Malfunctions

The function dbg:n/1 used a local fun to set up a tracer on a remote node. This works fine as long as the remote node is running exactly the same version of Erlang/OTP but does not work at all otherwise. This is fixed by exporting the relevant function and by calling this function on the remote node to set up remote tracing.

Own Id: OTP-16930 Aux Id: ERL-1371, GH-4396

Runtime_Tools 1.13.2

Improvements and New Features

Update of systemtap trace example scripts.

Own Id: OTP-15670

Runtime_Tools 1.13.1

Improvements and New Features

Optimize observer by using new system_info(ets_count) instead of more expensive length(ets:all()).

Own Id: OTP-15163 Aux Id: PR-1844

Runtime_Tools 1.13

Improvements and New Features

New utility module scheduler which makes it easier to measure scheduler utilization.

Own Id: OTP-14904

Runtime_Tools 1.12.5

Fixed Bugs and Malfunctions

system_information:to_file/1 will now use slightly less memory.

Own Id: OTP-14816

Runtime_Tools 1.12.4

Improvements and New Features

New family of erts_alloc strategies: Age Order First Fit. Similar to "address order", but instead the oldest possible carrier is always chosen for allocation.

Own Id: OTP-14917 Aux Id: ERIERL-88

Runtime_Tools 1.12.3

Fixed Bugs and Malfunctions

Removed all old unused files in the documentation.

Own Id: OTP-14475 Aux Id: ERL-409, PR-1493

Runtime_Tools 1.12.2

Improvements and New Features

General Unicode improvements.

Own Id: OTP-14462

Runtime_Tools 1.12.1

Fixed Bugs and Malfunctions

A faulty encoding comment was added when saving trace patterns to file. This is now corrected.

Own Id: OTP-14479

Runtime_Tools 1.12

Fixed Bugs and Malfunctions

Add compile option -compile(no_native) in modules with on_load directive which is not yet supported by HiPE.

Own Id: OTP-14316 Aux Id: PR-1390

Improvements and New Features

Miscellaneous updates due to atoms containing arbitrary Unicode characters.

Own Id: OTP-14285

Sockets can now be bound to device (SO_BINDTODEVICE) on platforms where it is supported.

This has been implemented e.g to support VRF-Lite under Linux; see VRF , and GitHub pull request #1326.

Own Id: OTP-14357 Aux Id: PR-1326

Runtime_Tools 1.11.1

Fixed Bugs and Malfunctions

etop erroneously reported the average scheduler utilization since the tool was first started instead of the scheduler utilization since last update. This is now corrected.

Own Id: OTP-14090 Aux Id: seq13232

Runtime_Tools 1.11

Improvements and New Features

Add option queue_size to ttb:tracer/2. This sets the maximum queue size for the IP trace driver which is used when tracing to shell and/or {local,File}.

The default value for queue_size is specified by dbg, and it is now changed from 50 to 200.

Own Id: OTP-13829 Aux Id: seq13171

The port information page is updated to show more information per port.

Own Id: OTP-13948 Aux Id: ERL-272

Runtime_Tools 1.10.1

Improvements and New Features

Correct some minor documentation issues.

Own Id: OTP-13891

Runtime_Tools 1.10

Fixed Bugs and Malfunctions

Fix bug in dbg:trace_port/2 that could cause the trace ip driver to produce faulty error reports "...(re)selected before stop_select was called for driver trace_ip_drv".

Own Id: OTP-13576 Aux Id: ERL-119

Improvements and New Features

Add microstate accounting

Microstate accounting is a way to track which state the different threads within ERTS are in. The main usage area is to pin point performance bottlenecks by checking which states the threads are in and then from there figuring out why and where to optimize.

Since checking whether microstate accounting is on or off is relatively expensive only a few of the states are enabled by default and more states can be enabled through configure.

There is a convenience module called msacc that has been added to runtime_tools that can assist in gathering and interpreting the data from Microstate accounting.

For more information see erlang:statistics(microstateaccounting, ) and the msacc module in runtime_tools.

Own Id: OTP-12345

Update observer GUI to support tracing on ports, and to set matchspecs for send/receive. This required some minor bugfixes in runtime_tools/dbg.

Own Id: OTP-13481

Update dbg and ttb to work with a tracer module as tracer and tracing on ports.

Own Id: OTP-13500

Updated dbg to accept the new trace options monotonic_timestamp and strict_monotonic_timestamp.

Own Id: OTP-13502

Introduce LTTng tracing via Erlang tracing.

For LTTng to be enabled OTP needs to be built with configure option --with-dynamic-trace=lttng.

The dynamic trace module dyntrace is now capable to be used as a LTTng sink for Erlang tracing. For a list of all tracepoints, see Runtime Tools User's Guide .

This feature also introduces an incompatible change in trace tags. The trace tags gc_start and gc_end has been split into gc_minor_start, gc_minor_end and gc_major_start, gc_major_end.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-13532

Runtime_Tools 1.9.3

Improvements and New Features

dbg:trace_client() now uses a read buffer to speed up reading of trace files.

Own Id: OTP-13279

Runtime_Tools 1.9.2

Improvements and New Features

Clarified dbg:stop documentation

Own Id: OTP-13078

Runtime_Tools 1.9.1

Fixed Bugs and Malfunctions

The trace_file_drv did not handle EINTR correct which caused it to fail when the runtime system received a signal.

Own Id: OTP-12890 Aux Id: seq12885

Runtime_Tools 1.9

Improvements and New Features

Change license text from Erlang Public License to Apache Public License v2

Own Id: OTP-12845

Runtime_Tools 1.8.16

Fixed Bugs and Malfunctions

The trace process started by dbg would not always terminate when dbg:stop/0 was called.

Own Id: OTP-12517

Runtime_Tools 1.8.15

Fixed Bugs and Malfunctions

Add nif_version to erlang:system_info/1 in order to get the NIF API version of the runtime system in a way similar to driver_version.

Own Id: OTP-12298

Runtime_Tools 1.8.14

Fixed Bugs and Malfunctions

The documentation for the return value of dbg:{stop,stop_clear} functions are now correct (Thanks to Luca Favatella)

Own Id: OTP-11603

Fix DTrace build on Illumos. (Thanks to Ryan Zezeski.)

Own Id: OTP-11622

Do not turn off scheduler_wall_time, as it can interfere with other applications usage.

Own Id: OTP-11693 Aux Id: seq12528

Application upgrade (appup) files are corrected for the following applications:

asn1, common_test, compiler, crypto, debugger, dialyzer, edoc, eldap, erl_docgen, et, eunit, gs, hipe, inets, observer, odbc, os_mon, otp_mibs, parsetools, percept, public_key, reltool, runtime_tools, ssh, syntax_tools, test_server, tools, typer, webtool, wx, xmerl

A new test utility for testing appup files is added to test_server. This is now used by most applications in OTP.

(Thanks to Tobias Schlager)

Own Id: OTP-11744

Improvements and New Features

Allow install path to have unicode characters.

Own Id: OTP-10877

The erts_alloc_config tool has been updated to produce configurations that better fit todays SMP support in the VM.

Own Id: OTP-11662

The app-file key runtime_dependencies has been introduced.

Runtime dependencies have been added to all app-files in OTP. Note that these may not be completely correct during OTP 17, but this is actively being worked on.

The function system_information:sanity_check/0 will verify all declared runtime dependencies in the system when called.

Own Id: OTP-11773

Runtime_Tools 1.8.13

Fixed Bugs and Malfunctions

Observer did not produce correct result when ERTS internal memory allocators had been disabled.

Own Id: OTP-11520

Runtime_Tools 1.8.12

Fixed Bugs and Malfunctions

The process trace flag 'silent' is now allowed in call to dbg:p/2.

Own Id: OTP-11222

Improvements and New Features

Introduced functionality for inspection of system and build configuration.

Own Id: OTP-11196

Runtime_Tools 1.8.11

Fixed Bugs and Malfunctions

Some bugs related to calculation of CPU/scheduler utilization in observer are corrected.

Current function for a process is accepted to be 'undefined' when running hipe.

Own Id: OTP-10894

Improvements and New Features

Erlang source files with non-ASCII characters are now encoded in UTF-8 (instead of latin1).

Own Id: OTP-11041 Aux Id: OTP-10907

Runtime_Tools 1.8.10

Fixed Bugs and Malfunctions

Fix Table Viewer refresh crash on no more existing ets tables (Thanks to Peti Gömori)

Own Id: OTP-10635

Improvements and New Features

User Guides for the dynamic tracing tools dtrace and systemtap have been added to the documentation.

Own Id: OTP-10155

Where necessary a comment stating encoding has been added to Erlang files. The comment is meant to be removed in Erlang/OTP R17B when UTF-8 becomes the default encoding.

Own Id: OTP-10630

Some examples overflowing the width of PDF pages have been corrected.

Own Id: OTP-10665

The backend module appmon_info.erl is moved from appmon application to runtime_tools. This allows appmon to be run from a remote erlang node towards a target node which does not have appmon (and its dependencies) installed, as long as runtime_tools is installed there.

Own Id: OTP-10786

Runtime_Tools 1.8.9

Fixed Bugs and Malfunctions

Change the module-level docs to give complete step-by-step instructions for using the `dyntrace:p()` trace function. (Thanks to Scott Lystig Fritchie)

Own Id: OTP-10141

Add 1024 separate USDT probes to dyntrace.erl and dyntrace.c (Thanks to Scott Lystig Fritchie)

Own Id: OTP-10143

Relocate bodies of DTrace probes to the statically-linked VM.

Due to various operating systems (in both the DTrace and SystemTap worlds) not fully supporting DTrace probes (or SystemTap-compatibility mode probes) in shared libraries, we relocate those probes to the statically-linked virtual machine. This could be seen as pollution of the pristine VM by a (yet) experimental feature. However:

1. This code can be eliminated completely by the C preprocessor. 2. Leaving the probes in the dyntrace NIF shared library simply does not work correctly on too many platforms. Many thanks to Macneil Shonle at Basho for assisting when my RSI-injured fingers gave out. (note: Solaris 10 and FreeBSD 9.0-RELEASE can take a long time to compile)

Own Id: OTP-10189

Runtime_Tools 1.8.8

Improvements and New Features

The DTrace source patch from Scott Lystig Fritchie is integrated in the source tree. Using an emulator with dtrace probe is still not supported for production use, but may be a valuable debugging tool. Configure with --with-dynamic-trace=dtrace (or --with-dynamic-trace=systemtap) to create a build with dtrace probes enabled. See runtime_tools for documentation and examples.

Own Id: OTP-10017

Runtime_Tools 1.8.7

Fixed Bugs and Malfunctions

Earlier dbg:stop only did erlang:trace_delivered and did not flush the trace file driver. Therefore there could still be trace messages that were delivered to the driver (guaranteed by erlang:trace_delivered) but not yet written to the file when dbg:stop returned. Flushing is now added on each node before the dbg process terminates.

Own Id: OTP-9651

File handles created by the trace_file_drv driver was inherited to child processes. This is now corrected.

Own Id: OTP-9658

Improvements and New Features

Erlang/OTP can now be built using parallel make if you limit the number of jobs, for instance using 'make -j6' or 'make -j10'. 'make -j' does not work at the moment because of some missing dependencies.

Own Id: OTP-9451

Two new built-in trace pattern aliases have been added: caller_trace (c) and caller_exception_trace (cx). See the dbg:ltp/0 documentation for more info.

Own Id: OTP-9458

Runtime_Tools 1.8.6

Improvements and New Features

Two new built-in trace pattern aliases have been added: caller_trace (c) and caller_exception_trace (cx). See the dbg:ltp/0 documentation for more info.

Own Id: OTP-9458

Runtime_Tools 1.8.5

Improvements and New Features

When a big number of trace patterns are set by inviso the Erlang VM could get unresponsive for several seconds. This is now corrected.

Own Id: OTP-9048 Aux Id: seq11480

Runtime_Tools 1.8.4.1

Fixed Bugs and Malfunctions

Minor corrections and removal of a temporary workaround.

Own Id: OTP-8755 Aux Id: seq-11628, seq-11639

Small fix in inviso_autostart_server.

Own Id: OTP-8783 Aux Id: seq11628

Runtime_Tools 1.8.4

Improvements and New Features

Miscellaneous updates.

Own Id: OTP-8705

Runtime_Tools 1.8.3

Improvements and New Features

Cross compilation improvements and other build system improvements.

Most notable:

Lots of cross compilation improvements. The old cross compilation support was more or less non-existing as well as broken. Please, note that the cross compilation support should still be considered as experimental. Also note that old cross compilation configurations cannot be used without modifications. For more information on cross compiling Erlang/OTP see the $ERL_TOP/INSTALL-CROSS.md file.

Support for staged install using DESTDIR. The old broken INSTALL_PREFIX has also been fixed. For more information see the $ERL_TOP/INSTALL.md file.

Documentation of the release target of the top Makefile. For more information see the $ERL_TOP/INSTALL.md file.

make install now by default creates relative symbolic links instead of absolute ones. For more information see the $ERL_TOP/INSTALL.md file.

$ERL_TOP/configure --help=recursive now works and prints help for all applications with configure scripts.

Doing make install, or make release directly after make all no longer triggers miscellaneous rebuilds.

Existing bootstrap system is now used when doing make install, or make release without a preceding make all.

The crypto and ssl applications use the same runtime library path when dynamically linking against libssl.so and libcrypto.so. The runtime library search path has also been extended.

The configure scripts of erl_interface and odbc now search for thread libraries and thread library quirks the same way as ERTS do.

The configure script of the odbc application now also looks for odbc libraries in lib64 and lib/64 directories when building on a 64-bit system.

The config.h.in file in the erl_interface application is now automatically generated in instead of statically updated which reduces the risk of configure tests without any effect.

(Thanks to Henrik Riomar for suggestions and testing)

(Thanks to Winston Smith for the AVR32-Linux cross configuration and testing)

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-8323

Cleanups suggested by tidier and modernization of types and specs.

Own Id: OTP-8455

Runtime_Tools 1.8.2

Improvements and New Features

The documentation is now built with open source tools (xsltproc and fop) that exists on most platforms. One visible change is that the frames are removed.

Own Id: OTP-8201

Runtime_Tools 1.8.1

Fixed Bugs and Malfunctions

Makefile.in has been updated to use the LDFLAGS environment variable (if set). (Thanks to Davide Pesavento.)

Own Id: OTP-8157

Runtime_Tools 1.8

Improvements and New Features

etop would crash if the emulator's custom allocators had been turned off (e.g. using the +Meamin option).

Own Id: OTP-7519

The copyright notices have been updated.

Own Id: OTP-7851

Now, dbg:p/2 accepts {X,Y,Z} process specification as stated in the documentation. It also now accepts "<X.Y.Z>" like from erlang:pid_to_list/1.

There is now a pre-saved match spec in dbg that saves a lot of typing. Use dbg:ltp/0 to find out more...

Own Id: OTP-7867

Runtime_Tools 1.7.3

Fixed Bugs and Malfunctions

Fixed a timestamp problem where some events could be sent out of order. Minor fixes to presentation of data.

Own Id: OTP-7544 Aux Id: otp-7442

Runtime_Tools 1.7.2

Fixed Bugs and Malfunctions

etop now collects process information faster and more reliably than before (a race condition reported by Igor Goryachev has been eliminated).

Trace messages could be lost when ttb:stop/0 was called.

Own Id: OTP-7164

Runtime_Tools 1.7.1

Improvements and New Features

The documentation has been updated so as to reflect the last updates of the Erlang shell as well as the minor modifications of the control sequence p of the io_lib module.

Superfluous empty lines have been removed from code examples and from Erlang shell examples.

Own Id: OTP-6944 Aux Id: OTP-6554, OTP-6911

Memory management improvements especially for the runtime system with SMP support:

The runtime system with SMP support can now use multiple, thread specific instances of most memory allocators. This improves performance since it reduces lock contention in the memory allocators. It may however increase memory usage for some applications. The runtime system with SMP support will by default enable this feature on most allocators. The amount of instances used can be configured.

driver_alloc(), driver_realloc(), and driver_free() now use their own erts specific memory allocator instead of the default malloc() implementation on the system.

The default configuration of some allocators have been changed to fit applications that use much memory better.

Some new erts_alloc configuration parameters have been added.

erts_alloc_config has been modified to be able to create configurations suitable for multiple instances of allocators.

The returned value from erlang:system_info({allocator, Alloc}) has been changed. This since an allocator may now run in multiple instances.

If you for some reason want the memory allocators to be configured as before, you can pass the +Mea r11b command-line argument to erl.

For more information see the erts_alloc(3), the m:erts_alloc_config, and the erlang documentation.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-7100

Runtime_Tools 1.7

Fixed Bugs and Malfunctions

dbg could leave traced processes in a suspended state if the tracer process was killed with exit reason kill.

erlang:suspend_process/2 has been introduced which accepts an option list as second argument. For more information see the erlang documentation.

Processes suspended via erlang:suspend_process/[1,2] will now be automatically resumed if the process that called erlang:suspend_process/[1,2] terminates.

Processes could previously be suspended by one process and resumed by another unless someone was tracing the suspendee. This is not possible anymore. The process resuming a process has to be the one that suspended it.

* POTENTIAL INCOMPATIBILITY *

Own Id: OTP-6946

Improvements and New Features

The undocumented and unsupported function dbg:tracer/1 has been removed. The undocumented, unsupported, and broken function dbg:i/1 has been removed.

Own Id: OTP-6939

Runtime_Tools 1.6.8

Fixed Bugs and Malfunctions

In this release the following has been fixed and enhanced: Autostart: It is now possible to configure modules that shall be loaded by the autostart mechanism. This because it is not certain that all application systems make use of the OTP boot script to set up paths to all Erlang modules. Runtime_tools/Inviso: A bug in the fetch_log functionality has been fixed. Further a bug that was (supposedly) fixed in a previous patch concerning meta-tracer write_ti has been fixed (again) in this patch. A bug in inviso_as_lib making remote autostart config file updates fail has been fixed. Inviso: inviso_tool has been given a flush API.

Own Id: OTP-6918

Runtime_Tools 1.6.7

Improvements and New Features

The following bugs/improvements have been done: Internal interworking between inviso_rt and inviso_rt_meta. The call function used by inviso_rt to call inviso_rt_meta is now protected with a monitor. Inviso_rt_meta now includes the timestamp of the incoming meta trace message when calling the call-callback. (Makes it possible to add a "better" timestamp to the ti-file.) Bug in inviso_tool making it not remove trace patterns when terminating. Bug in internal function h_start_session making inviso_tool crash if there were no active nodes to start the session on. The user-inviso_tool and inviso API-inviso control component request/response gen_server protocols had default time-out. Since many trace operations can be time consuming, a longer time-out is necessary. Improved overload protection. It is now possible to let the overload protection renew itself (e.g after an exit from an external overload protector). Inviso_rt_meta now fully uses the exception_trace match spec action term. Run Trace Case API (as in contrast to activate and deactivate trace case APIs) in inviso_tool. Flush trace-port API added to inviso. Get_session_data API added to inviso_tool. Improved inviso_tool:stop making it possible to name nodes which shall not have their trace patterns removed when inviso_tool terminates. Bug in handling of writing multiple ti-entries if returned from a call/return_from call-back in inviso_rt_meta Process trace flags are no longer explicitly removed by the inviso_tool when it terminates. Not necessary. Inviso_tool get_autostart_data adopted to standard autostarter.

* INCOMPATIBILITY with Meta trace call-backs are called with different arguments now. *

Own Id: OTP-6881

Runtime_Tools 1.6.6

Fixed Bugs and Malfunctions

A bug in inviso_rt_meta caused an ETS table containing information on initiated (init_tpm) functions to be lost when suspending tracing. Further an enhancement to inviso_rt has been introduced making it possible to activate process trace flags based on globally registered names. It is then not an error to activate a global name on a node where the name does not reside. The process count in the return value will simply be set to zero (hence exactly one node in the NodeResult part of the return value will indicate one matching process found). A bug was found in fetch_log API. At the same time the fetch_log functionality was enhanced to also offer flow control making fetcher processes send chunks of transferred file data at a slower pace.

Own Id: OTP-6703

Improvements and New Features

Minor Makefile changes.

Own Id: OTP-6689 Aux Id: OTP-6742

An experimental tool called erts_alloc_config has been added. erts_alloc_config is intended to be used to aid creation of an erts_alloc configuration that is suitable for a limited number of runtime scenarios. For more information see the m:erts_alloc_config documentation.

Own Id: OTP-6700

Runtime_Tools 1.6.5

Improvements and New Features

Misc improvements.

Own Id: OTP-6576

Runtime_Tools 1.6.4

Improvements and New Features

This application has been updated to eliminate warnings by Dialyzer.

Own Id: OTP-6551

Runtime_Tools 1.6.3

Fixed Bugs and Malfunctions

This ticket includes several improvements and bugfixes to both runtime_tools and inviso. The overload mechanism can now also react to incoming messages. This is useful if an external overload watch-dog is used. Some improvements of variable bindings has been done to the default autostart mechanism - inviso_autostart_server. Autostart "jobs" can now be done in parallel, allowing for some jobs to hang waiting for some parts of the traced system to become ready before proceeding. Previously when using named meta-match-specs (tpm_ms) ending up with zero match-specs still kept the meta trace pattern active. This caused zero match-specs to be equal to unlimited meta tracing on that particular function. If the internal database becomes empty of meta match specs, meta trace pattern is removed for that function. Standard public loop data in the inviso runtime meta tracer process is now extended to a 2-tuple. The functions ctp/1 and ctpl/1 are added making it possible to remove trace patterns for a list of functions rather than one by one. Inviso_rt_meta will now accept a list of binaries to be output into the trace information file, in additions to a single binary. Further it is also possible to make own output to the trace information file using the write_ti/1 function. An error was discovered in inviso_rt making the inviso_rt_meta remain rather than terminate if the inviso_rt terminated due to "running alone" (not allowed to run without a control component). A new tool, inviso_tool, has been added to the inviso application.

Own Id: OTP-6426

Runtime_Tools 1.6.2

Fixed Bugs and Malfunctions

Several minor bugs and race conditions eliminated in the runtime_tools and observer applications.

Own Id: OTP-6265

Runtime_Tools 1.6.1

Improvements and New Features

There are new BIFs erlang:spawn_monitor/1,3, and the new option monitor for spawn_opt/2,3,4,5.

The observer_backend module has been updated to handle the new BIFs.

Own Id: OTP-6281

Runtime_Tools 1.6

Added the runtime part of the Inviso tracer, see the new Inviso application for more information. This also meant adding an application callback module and an application supervisor tree for Runtime_Tools.

Runtime_Tools 1.5.1.1

Improvements and New Features

The dbg manual page has been updated with information about how to avoid deadlock when tracing.

Own Id: OTP-5373 Aux Id: seq9729

Runtime_Tools 1.5.1

Fixed Bugs and Malfunctions

Linked in drivers in the Crypto, and Asn1 applications are now compiled with the -D_THREAD_SAFE and -D_REENTRANT switches on unix when the emulator has thread support enabled.

Linked in drivers on MacOSX are not compiled with the undocumented -lbundle1.o switch anymore. Thanks to Sean Hinde who sent us a patch.

Linked in driver in Crypto, and port programs in SSL, now compiles on OSF1.

Minor Makefile improvements in Runtime_Tools.

Own Id: OTP-5346

LTTng and Erlang/OTP

Introduction

The Linux Trace Toolkit: next generation is an open source system software package for correlated tracing of the Linux kernel, user applications and libraries.

For more information, please visit http://lttng.org

Building Erlang/OTP with LTTng support

Configure and build Erlang with LTTng support:

For LTTng to work properly with Erlang/OTP you need the following packages installed:

LTTng-tools: a command line interface to control tracing sessions.

LTTng-UST: user space tracing library.

On Ubuntu this can be installed via aptitude:

$ sudo aptitude install lttng-tools liblttng-ust-dev

See Installing LTTng for more information on how to install LTTng on your system.

After LTTng is properly installed on the system Erlang/OTP can be built with LTTng support.

$ ./configure --with-dynamic-trace=lttng
$ make

Dyntrace Tracepoints

All tracepoints are in the domain of org_erlang_dyntrace

All Erlang types are the string equivalent in LTTng.

process_spawn

pid : string :: Process ID. Ex. "<0.131.0>"

parent : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

Available through erlang:trace/3 with trace flag procs and {tracer,dyntrace,[]} as tracer module.

Example:

process_spawn: { cpu_id = 3 }, { pid = "<0.131.0>", parent = "<0.130.0>", entry = "erlang:apply/2" }

process_link

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

from : string :: Process ID or Port ID. Ex. "<0.131.0>"

type : string :: "link" | "unlink"

Available through erlang:trace/3 with trace flag procs and {tracer,dyntrace,[]} as tracer module.

Example:

process_link: { cpu_id = 3 }, { from = "<0.130.0>", to = "<0.131.0>", type = "link" }

process_exit

pid : string :: Process ID. Ex. "<0.131.0>"

reason : string :: Exit reason. Ex. "normal"

Available through erlang:trace/3 with trace flag procs and {tracer,dyntrace,[]} as tracer module.

Example:

process_exit: { cpu_id = 3 }, { pid = "<0.130.0>", reason = "normal" }

process_register

pid : string :: Process ID. Ex. "<0.131.0>"

name : string :: Registered name. Ex. "logger"

type : string :: "register" | "unregister"

Example:

process_register: { cpu_id = 0 }, { pid = "<0.128.0>", name = "dyntrace_lttng_SUITE" type = "register" }

process_scheduled

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

type : string :: "in" | "out" | "in_exiting" | "out_exiting" | "out_exited"

Available through erlang:trace/3 with trace flag running and {tracer,dyntrace,[]} as tracer module.

Example:

process_scheduled: { cpu_id = 0 }, { pid = "<0.136.0>", entry = "erlang:apply/2", type = "in" }

port_open

pid : string :: Process ID. Ex. "<0.131.0>"

driver : string :: Driver name. Ex. "tcp_inet"

port : string :: Port ID. Ex. "#Port<0.1031>"

Available through erlang:trace/3 with trace flag ports and {tracer,dyntrace,[]} as tracer module.

Example:

port_open: { cpu_id = 5 }, { pid = "<0.131.0>", driver = "'/bin/sh -s unix:cmd'", port = "#Port<0.1887>" }

port_exit

port : string :: Port ID. Ex. "#Port<0.1031>"

reason : string :: Exit reason. Ex. "normal"

Available through erlang:trace/3 with trace flag ports and {tracer,dyntrace,[]} as tracer module.

Example:

port_exit: { cpu_id = 5 }, { port = "#Port<0.1887>", reason = "normal" }

port_link

to : string :: Process ID. Ex. "<0.131.0>"

from : string :: Process ID. Ex. "<0.131.0>"

type : string :: "link" | "unlink"

Available through erlang:trace/3 with trace flag ports and {tracer,dyntrace,[]} as tracer module.

Example:

port_link: { cpu_id = 5 }, { from = "#Port<0.1887>", to = "<0.131.0>", type = "unlink" }

port_scheduled

Available through erlang:trace/3 with trace flag running and {tracer,dyntrace,[]} as tracer module.

port : string :: Port ID. Ex. "#Port<0.1031>"

entry : string :: Callback. Ex. "open"

type : string :: "in" | "out" | "in_exiting" | "out_exiting" | "out_exited"

Example:

port_scheduled: { cpu_id = 5 }, { pid = "#Port<0.1905>", entry = "close", type = "out" }

Available through erlang:trace/3 with trace flag running and {tracer,dyntrace,[]} as tracer module.

function_call

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

depth : integer :: Stack depth. Ex. 0

Available through erlang:trace/3 with trace flag call and {tracer,dyntrace,[]} as tracer module.

Example:

function_call: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:'-t_call/1-fun-1-'/0", depth = 0 }

function_return

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

depth : integer :: Stack depth. Ex. 0

Available through erlang:trace/3 with trace flag call or return_to and {tracer,dyntrace,[]} as tracer module.

Example:

function_return: { cpu_id = 5 }, { pid = "<0.145.0>", entry = "dyntrace_lttng_SUITE:waiter/0", depth = 0 }

function_exception

pid : string :: Process ID. Ex. "<0.131.0>"

entry : string :: Code Location. Ex. "lists:sort/1"

class : string :: Error reason. Ex. "error"

Available through erlang:trace/3 with trace flag call and {tracer,dyntrace,[]} as tracer module.

Example:

function_exception: { cpu_id = 5 }, { pid = "<0.144.0>", entry = "t:call_exc/1", class = "error" }

message_send

from : string :: Process ID or Port ID. Ex. "<0.131.0>"

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

message : string :: Message sent. Ex. "{<0.162.0>,ok}"

Available through erlang:trace/3 with trace flag send and {tracer,dyntrace,[]} as tracer module.

Example:

message_send: { cpu_id = 3 }, { from = "#Port<0.1938>", to = "<0.160.0>", message = "{#Port<0.1938>,eof}" }

message_receive

to : string :: Process ID or Port ID. Ex. "<0.131.0>"

message : string :: Message received. Ex. "{<0.162.0>,ok}"

Available through erlang:trace/3 with trace flag 'receive' and {tracer,dyntrace,[]} as tracer module.

Example:

message_receive: { cpu_id = 7 }, { to = "<0.167.0>", message = "{<0.165.0>,ok}" }

gc_minor_start

pid : string :: Process ID. Ex. "<0.131.0>"

need : integer :: Heap need. Ex. 2

heap : integer :: Young heap word size. Ex. 233

old_heap : integer :: Old heap word size. Ex. 233

Available through erlang:trace/3 with trace flag garbage_collection and {tracer,dyntrace,[]} as tracer module.

Example:

gc_minor_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 0, heap = 610, old_heap = 0 }

gc_minor_end

pid : string :: Process ID. Ex. "<0.131.0>"

reclaimed : integer :: Heap reclaimed. Ex. 2

heap : integer :: Young heap word size. Ex. 233

old_heap : integer :: Old heap word size. Ex. 233

Available through erlang:trace/3 with trace flag garbage_collection and {tracer,dyntrace,[]} as tracer module.

Example:

gc_minor_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 120, heap = 1598, old_heap = 1598 }

gc_major_start

pid : string :: Process ID. Ex. "<0.131.0>"

need : integer :: Heap need. Ex. 2

heap : integer :: Young heap word size. Ex. 233

old_heap : integer :: Old heap word size. Ex. 233

Available through erlang:trace/3 with trace flag garbage_collection and {tracer,dyntrace,[]} as tracer module.

Example:

gc_major_start: { cpu_id = 0 }, { pid = "<0.172.0>", need = 8, heap = 2586, old_heap = 1598 }

gc_major_end

pid : string :: Process ID. Ex. "<0.131.0>"

reclaimed : integer :: Heap reclaimed. Ex. 2

heap : integer :: Young heap word size. Ex. 233

old_heap : integer :: Old heap word size. Ex. 233

Available through erlang:trace/3 with trace flag garbage_collection and {tracer,dyntrace,[]} as tracer module.

Example:

gc_major_end: { cpu_id = 0 }, { pid = "<0.172.0>", reclaimed = 240, heap = 4185, old_heap = 0 }

BEAM Tracepoints

All tracepoints are in the domain of org_erlang_otp

All Erlang types are the string equivalent in LTTng.

driver_init

driver : string :: Driver name. Ex. "tcp_inet"

major : integer :: Major version. Ex. 3

minor : integer :: Minor version. Ex. 1

flags : integer :: Flags. Ex. 1

Example:

driver_init: { cpu_id = 2 }, { driver = "caller_drv", major = 3, minor = 3, flags = 1 }

driver_start

pid : string :: Process ID. Ex. "<0.131.0>"

driver : string :: Driver name. Ex. "tcp_inet"

port : string :: Port ID. Ex. "#Port<0.1031>"

Example:

driver_start: { cpu_id = 2 }, { pid = "<0.198.0>", driver = "caller_drv", port = "#Port<0.3676>" }

driver_output

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

bytes : integer :: Size of data returned. Ex. 82

Example:

driver_output: { cpu_id = 2 }, { pid = "<0.198.0>", port = "#Port<0.3677>", driver = "/bin/sh -s unix:cmd", bytes = 36 }

driver_outputv

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

bytes : integer :: Size of data returned. Ex. 82

Example:

driver_outputv: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet", bytes = 3 }

driver_ready_input

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_ready_input: { cpu_id = 5 }, { pid = "<0.189.0>", port = "#Port<0.3637>", driver = "inet_gethost 4 " }

driver_ready_output

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_ready_output: { cpu_id = 5 }, { pid = "<0.194.0>", port = "#Port<0.3663>", driver = "tcp_inet" }

driver_timeout

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_timeout: { cpu_id = 5 }, { pid = "<0.196.0>", port = "#Port<0.3664>", driver = "tcp_inet" }

driver_stop_select

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_stop_select: { cpu_id = 5 }, { driver = "unknown" }

driver_flush

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_flush: { cpu_id = 7 }, { pid = "<0.204.0>", port = "#Port<0.3686>", driver = "tcp_inet" }

driver_stop

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_stop: { cpu_id = 5 }, { pid = "[]", port = "#Port<0.3673>", driver = "tcp_inet" }

driver_process_exit

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

driver_ready_async

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

Example:

driver_ready_async: { cpu_id = 3 }, { pid = "<0.181.0>", port = "#Port<0.3622>", driver = "tcp_inet" }

driver_call

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

command : integer :: Command integer. Ex. 1

bytes : integer :: Size of data returned. Ex. 82

Example:

driver_call: { cpu_id = 2 }, { pid = "<0.202.0>", port = "#Port<0.3676>", driver = "caller_drv", command = 0, bytes = 2 }

driver_control

pid : string :: Process ID. Ex. "<0.131.0>"

port : string :: Port ID. Ex. "#Port<0.1031>"

driver : string :: Driver name. Ex. "tcp_inet"

command : integer :: Command integer. Ex. 1

bytes : integer :: Size of data returned. Ex. 82

Example:

driver_control: { cpu_id = 3 }, { pid = "<0.32767.8191>", port = "#Port<0.0>", driver = "forker", command = 83, bytes = 32 }

carrier_create

type : string :: Carrier type. Ex. "ets_alloc"

instance : integer :: Allocator instance. Ex. 1

size : integer :: Carrier size. Ex. 262144

mbc_carriers : integer :: Number of multiblock carriers in instance. Ex. 3

mbc_carriers_size : integer :: Total size of multiblock blocks carriers in instance. Ex. 1343488

mbc_blocks : integer :: Number of multiblock blocks in instance. Ex. 122

mbc_blocks_size : integer :: Total size of all multiblock blocks in instance. Ex. 285296

sbc_carriers : integer :: Number of singleblock carriers in instance. Ex. 1

sbc_carriers_size : integer :: Total size of singleblock blocks carriers in instance. Ex. 1343488

sbc_blocks : integer :: Number of singleblocks in instance. Ex. 1

sbc_blocks_size : integer :: Total size of all singleblock blocks in instance. Ex. 285296

Example:

carrier_create: { cpu_id = 2 }, { type = "ets_alloc", instance = 7, size = 2097152, mbc_carriers = 4, mbc_carriers_size = 3440640, mbc_blocks = 526, mbc_blocks_size = 1278576, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }

carrier_destroy

type : string :: Carrier type. Ex. "ets_alloc"

instance : integer :: Allocator instance. Ex. 1

size : integer :: Carrier size. Ex. 262144

mbc_carriers : integer :: Number of multiblock carriers in instance. Ex. 3

mbc_carriers_size : integer :: Total size of multiblock blocks carriers in instance. Ex. 1343488

mbc_blocks : integer :: Number of multiblock blocks in instance. Ex. 122

mbc_blocks_size : integer :: Total size of all multiblock blocks in instance. Ex. 285296

sbc_carriers : integer :: Number of singleblock carriers in instance. Ex. 1

sbc_carriers_size : integer :: Total size of singleblock blocks carriers in instance. Ex. 1343488

sbc_blocks : integer :: Number of singleblocks in instance. Ex. 1

sbc_blocks_size : integer :: Total size of all singleblock blocks in instance. Ex. 285296

Example:

carrier_destroy: { cpu_id = 6 }, { type = "ets_alloc", instance = 7, size = 262144, mbc_carriers = 3, mbc_carriers_size = 3178496, mbc_blocks = 925, mbc_blocks_size = 2305336, sbc_carriers = 0, sbc_carriers_size = 0, sbc_blocks = 0, sbc_blocks_size = 0 }

carrier_pool_put

type : string :: Carrier type. Ex. "ets_alloc"

instance : integer :: Allocator instance. Ex. 1

size : integer :: Carrier size. Ex. 262144

Example:

carrier_pool_put: { cpu_id = 3 }, { type = "ets_alloc", instance = 5, size = 1048576 }

carrier_pool_get

type : string :: Carrier type. Ex. "ets_alloc"

instance : integer :: Allocator instance. Ex. 1

size : integer :: Carrier size. Ex. 262144

Example:

carrier_pool_get: { cpu_id = 7 }, { type = "ets_alloc", instance = 4, size = 3208 }

Example of process tracing

An example of process tracing of os_mon and friends.

Clean start of lttng in a bash shell.

$ lttng create erlang-demo
Spawning a session daemon
Session erlang-demo created.
Traces will be written in /home/egil/lttng-traces/erlang-demo-20160526-165920

Start an Erlang node with lttng enabled.

$ erl
Erlang/OTP 19 [erts-8.0] [source-4d7b24d] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [lttng]

Eshell V8.0  (abort with ^G)
1>

Load the dyntrace module.

1> l(dyntrace).
{module,dyntrace}

All tracepoints via dyntrace are now visible and can be listed through lttng list -u.

Enable the process_register LTTng tracepoint for Erlang.

$ lttng enable-event -u org_erlang_dyntrace:process_register
UST event org_erlang_dyntrace:process_register created in channel channel0

Enable process tracing for new processes and use dyntrace as tracer backend.

2> erlang:trace(new,true,[procs,{tracer,dyntrace,[]}]).
0

Start LTTng tracing.

$ lttng start
Tracing started for session erlang-demo

Start the os_mon application in Erlang.

3> application:ensure_all_started(os_mon).
{ok,[sasl,os_mon]}

Stop LTTng tracing and view the result.

$ lttng stop
Tracing stopped for session erlang-demo
$ lttng view
[17:20:42.561168759] (+?.?????????) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.66.0>", name = "sasl_sup", type = "register" }
[17:20:42.561215519] (+0.000046760) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.67.0>", name = "sasl_safe_sup", type = "register" }
[17:20:42.562149024] (+0.000933505) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.68.0>", name = "alarm_handler", type = "register" }
[17:20:42.571035803] (+0.008886779) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.69.0>", name = "release_handler", type = "register" }
[17:20:42.574939868] (+0.003904065) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.74.0>", name = "os_mon_sup", type = "register" }
[17:20:42.576818712] (+0.001878844) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.75.0>", name = "disksup", type = "register" }
[17:20:42.580032013] (+0.003213301) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.76.0>", name = "memsup", type = "register" }
[17:20:42.583046339] (+0.003014326) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.78.0>", name = "cpu_sup", type = "register" }
[17:20:42.586206242] (+0.003159903) elxd1168lx9 org_erlang_dyntrace:process_register: \
    { cpu_id = 5 }, { pid = "<0.82.0>", name = "timer_server", type = "register" }

DTrace and Erlang/OTP

History

The first implementation of DTrace probes for the Erlang virtual machine was presented at the 2008 Erlang User Conference. That work, based on the Erlang/OTP R12 release, was discontinued due to what appears to be miscommunication with the original developers.

Several users have created Erlang port drivers, linked-in drivers, or NIFs that allow Erlang code to try to activate a probe, e.g. foo_module:dtrace_probe("message goes here!").

Goals

Annotate as much of the Erlang VM as is practical.

The initial goal is to trace file I/O operations.

Support all platforms that implement DTrace: OS X, Solaris, and (I hope) FreeBSD and NetBSD.

To the extent that it's practical, support SystemTap on Linux via DTrace provider compatibility.

Allow Erlang code to supply annotations.

Supported platforms

OS X 10.6.x / Snow Leopard, OS X 10.7.x / Lion and probably newer versions.

Solaris 10. I have done limited testing on Solaris 11 and OpenIndiana release 151a, and both appear to work.

FreeBSD 9.0 and 10.0.

Linux via SystemTap compatibility. Please see $ERL_TOP/HOWTO/SYSTEMTAP.md for more details.

Just add the --with-dynamic-trace=dtrace option to your command when you run the configure script. If you are using systemtap, the configure option is --with-dynamic-trace=systemtap

Status

As of R15B01, the dynamic trace code is included in the OTP source distribution, although it's considered experimental. The main development of the dtrace code still happens outside of Ericsson, but there is no need to fetch a patched version of the OTP source to get the basic functionality.

DTrace probe specifications

Probe specifications can be found in erts/emulator/beam/erlang_dtrace.d, and a few example scripts can be found under lib/runtime_tools/examples/.

SystemTap and Erlang/OTP

Introduction

SystemTap is DTrace for Linux. In fact Erlang's SystemTap support is built using SystemTap's DTrace compatibility's layer. For an introduction to Erlang DTrace support read $ERL_TOP/HOWTO/DTRACE.md.

Requisites

Linux Kernel with UTRACE support

check for UTRACE support in your current kernel:

# grep CONFIG_UTRACE /boot/config-`uname -r`
CONFIG_UTRACE=y

Fedora 16 is known to contain UTRACE, for most other Linux distributions a custom build kernel will be required. Check Fedora's SystemTap documentation for additional required packages (e.g. Kernel Debug Symbols)

SystemTap > 1.6

A the time of writing this, the latest released version of SystemTap is version 1.6. Erlang's DTrace support requires a MACRO that was introduced after that release. So either get a newer release or build SystemTap from git yourself (see: http://sourceware.org/systemtap/getinvolved.html)

Building Erlang

Configure and build Erlang with SystemTap support:

# ./configure --with-dynamic-trace=systemtap + whatever args you need
# make

Testing

SystemTap, unlike DTrace, needs to know what binary it is tracing and has to be able to read that binary before it starts tracing. Your probe script therefore has to reference the correct beam emulator and stap needs to be able to find that binary. The examples are written for "beam", but other versions such as "beam.smp" or "beam.debug.smp" might exist (depending on your configuration). Make sure you either specify the full the path of the binary in the probe or your "beam" binary is in the search path.

All available probes can be listed like this:

# stap -L 'process("beam").mark("*")'

or:

# PATH=/path/to/beam:$PATH stap -L 'process("beam").mark("*")'

Probes in the dtrace.so NIF library like this:

# PATH=/path/to/dtrace/priv/lib:$PATH stap -L 'process("dtrace.so").mark("*")'

Running SystemTap scripts

Adjust the process("beam") reference to your beam version and attach the script to a running "beam" instance:

# stap /path/to/probe/script/port1.systemtap -x <pid of beam>

Tracing in Erlang with dbg

The dbg module in Erlang provides a text-based interface for tracing function calls, processes, ports, and messages. It simplifies the use of the underlying trace:process/4, trace:port/4, and trace:function/4 BIFs (Built-In Functions). This guide will walk you through the basics of using dbg for your Erlang applications.

This facility is useful for both quick debugging sessions in the shell and for more structured system testing, especially where other tools might have too much performance impact.

Quick Start

To trace a call to a function with minimal fuss, call dbg:c(Module, Name, Arguments). It starts a temporary trace receiver, enables all trace flags, and calls the designated function from a temporary process. For example, here is how to trace a call to application:which_applications/0:

1> dbg:c(application, which_applications, []).
(<0.92.0>) <0.45.0> ! {'$gen_call',{<0.92.0>,
                                    [alias|
                                     #Ref<0.0.11779.270031856.1478295555.230456>]},
                                   which_applications} (Timestamp: {1710,
                                                                    847802,
                                                                    479222})
(<0.92.0>) out {gen,do_call,4} (Timestamp: {1710,847802,479231})
(<0.92.0>) in {gen,do_call,4} (Timestamp: {1710,847802,479271})
(<0.92.0>) << {[alias|#Ref<0.0.11779.270031856.1478295555.230456>],
               [{stdlib,"ERTS  CXC 138 10","5.2.1"},
                {kernel,"ERTS  CXC 138 10","9.2.2"}]} (Timestamp: {1710,
                                                                   847802,
                                                                   479274})
[{stdlib,"ERTS  CXC 138 10","5.2.1"},
 {kernel,"ERTS  CXC 138 10","9.2.2"}]

In this example, four trace events are generated:

A send event (!) for the sending of a request from the current process to the application_controller process.

A schedule-out event (out) when the current process schedules out while waiting in a receive for the reply to arrive.

A schedule-in event (in) when the current process is scheduled in when reply has arrived.

A receive event (<<) when the current process retrieves the reply from the application_controller process.

The dbg:c/4 function has a fourth argument for specifying the trace flags, (see flags).

How-to Trace Systems

For more control, another way of tracing is to explicitly start a tracer and set the trace flags of your choice on the processes you want to trace. This is useful, when there is a complex system of processes, ports or nodes interacting where dbg:c/3 is to blunt.

Starting a Tracer (dbg:tracer/0,2)

First, you need to start a tracer process that will receive and display trace messages.

1> dbg:tracer().  % Start the default trace message receiver
{ok,<0.90.0>}     % <0.90.0> is the PID of the tracer process

This starts a server on the local node that will be the recipient of all trace messages. It uses a default handler that prints formatted trace messages to the Erlang shell.

If you need a custom tracer other than the default, you can create a tracer using dbg:tracer(Type, Data):

Type = process: Data is {HandlerFun, InitialState}. HandlerFun is a fun/2 that takes the trace message and the previous state, returning a new state.

Type = port: Data is a fun/0 that returns a new trace port (e.g., created by dbg:trace_port/2).

Type = module: Data is {TracerModule, TracerState} or a fun/0 returning this, for use with erl_tracer.

Type = file: Data is a filename where traces will be printed.

Note that only one tracer using this method can be started at a time. You can use trace sessions to start multiple tracers (see trace sessions).

Tracing Processes and Ports (dbg:p/1,2)

Once the tracer is started, you can tell dbg which processes or ports to trace and what events to trace for them using dbg:p(Item, Flags).

Item can be:

pid/0 or port/0 - The corresponding process or port is traced. The process or port can be a remote process or port (on another Erlang node). The node must be in the list of traced nodes (see dbg:n/1 and dbg:tracer/3).

all - All processes and ports in the system as well as all processes and ports created hereafter are to be traced.

processes - All processes in the system as well as all processes created hereafter are to be traced.

ports - All ports in the system as well as all ports created hereafter are to be traced.

new - All processes and ports created after the call are to be traced.

new_processes - All processes created after the call are to be traced.

new_ports - All ports created after the call are to be traced.

existing - All existing processes and ports are traced.

existing_processes - All existing processes are traced.

existing_ports - All existing ports are traced.

atom/0 - The process or port with the corresponding registered name is traced. The process or port can on another Erlang node. The node must be in the list of traced nodes (see dbg:n/1 and dbg:tracer/3).

integer/0 - The process <0.Item.0> is traced.

{X, Y, Z} - The process <X.Y.Z> is traced.

string/0 - If the Item is a string "<X.Y.Z>" as returned from pid_to_list/1, the process <X.Y.Z> is traced.

Flags can be a single atom or a list of flags. The available flags are:

s (send) - Traces the messages the process or port sends.

r (receive) - Traces the messages the process or port receives.

m (messages) - Traces the messages the process or port receives and sends.

c (call) - Traces global function calls for the process according to the trace patterns set in the system (see dbg:tp/2).

p (procs) - Traces process related events to the process.

ports - Traces port related events to the port.

sos (set on spawn) - Lets all processes created by the traced process inherit the trace flags of the traced process.

sol (set on link) - Lets another process, P2, inherit the trace flags of the traced process whenever the traced process links to P2.

sofs (set on first spawn) - This is the same as sos, but only for the first process spawned by the traced process.

sofl (set on first link) - This is the same as sol, but only for the first call to link/1 by the traced process.

all - Sets all flags except silent.

clear - Clears all flags.

Other flags accepted by trace:process/4 or trace:port/4 (e.g., timestamp, arity, return_to).

dbg:p(Item) is a shorthand for dbg:p(Item, [m]).

This function returns either an error tuple or an {ok, List} tuple. The List consists of specifications of how many processes and ports that matched (in the case of a single pid exactly 1). The specification of matched processes is {matched, Node, N}. If the remote processor call (using rpc) to a remote node fails, the rpc error message is returned as the fourth element in the tuple and the number of matched processes is 0.

Example: Trace messages and process events for a specific process

1> Pid = spawn(fun() -> receive {From,Msg} -> From ! Msg end end).
<0.90.0>
2> dbg:tracer().
{ok,<0.92.0>}
3> dbg:p(Pid, [m,procs]). % Trace messages and process events for Pid
{ok,[{matched,nonode@nohost,1}]}
4> Pid ! {self(),hello}.
(<0.90.0>) << {<0.88.0>,hello} % Received by Pid
{<0.88.0>,hello}
(<0.90.0>) <0.88.0> ! hello    % Sent by Pid
(<0.90.0>) exit normal         % Process event: Pid exited
5> flush().
Shell got hello
ok

Tracing Function Calls (dbg:tp/2,3,4, dbg:tpl/2,3,4)

To trace function calls, you need to:

Enable the c/call flag for the process(es) that will make the calls (using dbg:p/2).

Set a trace pattern for the function(s) you want to trace using dbg:tp/2 or dbg:tpl/2.

tp stands for trace pattern (for exported functions by default). tpl stands for trace pattern local (for local or remote calls to local and exported functions).

The general syntax is dbg:tp(ModuleOrMFA, MatchSpec) or dbg:tpl(ModuleOrMFA, MatchSpec).

ModuleOrMFA can be:

Module :: atom(): Equivalent to {Module, '_', '_'} (trace all functions in the module).

{Module, Function, Arity}: Trace specific function. '_' can be used as a wildcard.

Note that if the Module is specified as '_', the Function and Arity parts must be specified as '_' as well. The same holds for the Function in relation to Arity.

MatchSpec defines what to trace and how, see match specifications.

For simple call tracing, you can insert the empty list [].

The most common generic match specifications used can be found as built-in aliases.

x or exception_trace: Shows function names, parameters, return values, and exceptions. [{'_',[],[{exception_trace}]}]

c or caller_trace: Shows function names, parameters, and caller information. [{'_',[],[{message,{caller_line}}]}]

cx or caller_exception_trace: Combines x and c. [{'_',[],[{exception_trace},{message,{caller_line}}]}]

Example using built-in aliases:

1> dbg:tracer().
{ok,<0.90.0>}
2> dbg:p(all, c). % Short for dbg:p(all, call)
{ok,[{matched,nonode@nohost,49}]}
3> dbg:tp(lists, seq, cx). % cx: call and exception tracing with caller info
{ok,[{matched,nonode@nohost,2},{saved,cx}]}
4> lists:seq(1, 3).
(<0.88.0>) call lists:seq(1,3) ({erl_eval,do_apply,7,{"erl_eval.erl",904}})
[1,2,3]
(<0.88.0>) returned from lists:seq/2 -> [1,2,3]

Note that the caller info is the function that called lists:seq with file and line number.

Tracing Message Events (dbg:tpe/2)

By default, if send or receive tracing is enabled for a process, all such events are traced. dbg:tpe(Event, MatchSpec) allows you to filter these events.

Event: send or 'receive'.

MatchSpec: A match specifications.

For send: Matches on [Receiver, Msg].

For 'receive': Matches on [Node, Sender, Msg].

Managing Trace Patterns

You can display, remove, save and load trace pattern matchspecifications, if any of these bullets are of interest, click the title to read the documentation.

dbg:ltp() (List Trace Patterns): Lists all saved match specifications (from previous tp calls where a pattern was complex enough to be saved) and built-in aliases.

dbg:dtp() (Delete Trace Patterns): Deletes all saved (not built-in) match specifications.

dbg:dtp(N) (Delete Trace Pattern N): Deletes a specific saved pattern by its ID N.

dbg:wtp(FileName) (Write Trace Patterns): Saves current (saved and built-in) match specifications to FileName.

dbg:rtp(FileName) (Read Trace Patterns): Reads match specifications from FileName and merges them.

To stop tracing specific functions, you clear their trace patterns.

dbg:ctp(ModuleOrMFA): Clears both global and local trace patterns.

ctp(): Clears all trace patterns for all functions.

ctp(Module): Clears patterns for all functions in Module.

ctp(Module, Function) / ctp(Module, Function, Arity): More specific.

dbg:ctpl(ModuleOrMFA): Clears only local trace patterns (set by tpl).

dbg:ctpg(ModuleOrMFA): Clears only global trace patterns (set by tp).

dbg:ctpe(Event): Clears the match specification for send or 'receive', reverting to tracing all such events if the flag is set.

Match Specifications

Match Specifications are a powerful mini-language used to define conditions for tracing and actions to take. The dbg:tp/2, dbg:tpl/2 and dbg:tpe/2 functions accept them. For a description of the format for the MatchSpec argument, see Match Specifications in Erlang, which explains the general match specification language. For most users, dbg:fun2ms/1 explained below will do.

A match specification is a list of tuples: [{MatchHead, Guard, BodyActions}].

MatchHead: Patterns to match function arguments. '_' matches anything. '$1' is a variable.

Guard: Conditions that must be true.

BodyActions: Actions like {return_trace} (trace return value), {message, term()} (include extra info), {set_seq_token, ...}.

Creating Match Specifications with dbg:fun2ms/1

You can use dbg:fun2ms/1 to translate a literal Erlang fun into a match specification. This often feels more natural than writing the raw match spec. The fun must take a single list argument (matching the function arguments) and its body can use guard expressions and special tracing functions. The parse transform module ms_transform must be enabled. The easiest way to enable it is by adding the following line to the source file: -include_lib("stdlib/include/ms_transform.hrl"). In the shell its already enabled.

The head of the fun must be a single pattern that matches a list. That pattern will be used to match the arguments for the call:

1> dbg:fun2ms(fun([_,_]) -> true end). % Matches a function with two arguments
[{['_','_'],[],[true]}]
2> dbg:fun2ms(fun([A]) when is_atom(A) -> return_trace() end).
[{['$1'],[{is_atom,'$1'}],[{return_trace}]}]

The first match specification matches when a function having two arguments is called. The second matches when a function, taking one atom as an argument, is called.

Trace Sessions

To avoid interference between different tracing activities, you can create isolated dbg sessions.

First you create a session with dbg:session_create(Name) where the name is an atom, a session/0 is returned. Several sessions may have the same name.

When you have the session/0, you use dbg:session(Session, Fun). This function runs dbg commands within Fun using the specified session.

Any dbg function that is called with in the provided fun will use the session/0 provided instead of the default dbg session. This means that the tracing will be isolated from other tracing users on the system.

When you no longer need the session, use dbg:session_destroy(Session).

Example:

1> S = dbg:session_create(my_session).
<0.91.0>
2> dbg:session(S, fun() -> dbg:tracer(), dbg:p(all,c), dbg:tp(lists,seq,x) end).
{ok,[{matched,nonode@nohost,2},{saved,x}]}
3> lists:seq(1, 10).
(<0.89.0>) call lists:seq(1,10)
(<0.89.0>) returned from lists:seq/2 -> [1,2,3,4,5,6,7,8,9,10]
[1,2,3,4,5,6,7,8,9,10]
4> dbg:session_destroy(S).
ok

The state of the session/0 is preserved in between dbg:session/2 calls, so you can call dbg:session/2 multiple times when debugging you application.

Example:

1> S = dbg:session_create(my_session).
<0.91.0>
%% Setup the initial traces
2> dbg:session(S, fun() -> dbg:tracer(), dbg:p(self(),c), dbg:tp(lists,seq,x) end).
{ok,[{matched,nonode@nohost,2},{saved,x}]}
3> lists:seq(1, 3).
(<0.89.0>) call lists:seq(1,3)
(<0.89.0>) returned from lists:seq/2 -> [1,2,3]
[1,2,3]
%% Add an additional trace pattern
4> dbg:session(S, fun() -> dbg:tpl(lists,seq_loop,x) end).
ok
5> lists:seq(1, 3).
(<0.89.0>) call lists:seq(1,3)
(<0.89.0>) call lists:seq_loop(3,3,[])
(<0.89.0>) call lists:seq_loop(1,1,[2,3])
(<0.89.0>) returned from lists:seq_loop/3 -> [1,2,3]
(<0.89.0>) returned from lists:seq_loop/3 -> [1,2,3]
(<0.89.0>) returned from lists:seq/2 -> [1,2,3]
[1,2,3]
6> dbg:session_destroy(S).
ok

Trace on Remote Nodes

The dbg server keeps a list of nodes where tracing should be performed. Whenever a dbg:tp/2 call or a dbg:p/2 call is made, it is executed for all nodes in this list including the local node (except for dbg:p/2 with a specific pid/0 or port/0 as first argument, in which case the command is executed only on the node where the designated process or port resides).

dbg:n(Nodename): When this function is called, it starts a tracer process on the remote node, which will send all trace messages to the tracer process on the local node (via the Erlang distribution). If no tracer process is running on the local node, the error reason no_local_tracer is returned. The tracer process on the local node must be started with the dbg:tracer/0,2 function.

If Nodename is the local node, the error reason cant_add_local_node is returned. The function will also return an error if the node Nodename is not reachable.

If a trace port (see dbg:trace_port/2) is running on the local node, remote nodes cannot be traced with a tracer process. The error reason cant_trace_remote_pid_to_local_port is returned. However, a trace port can be started on the remote node with the dbg:tracer/3 function.

dbg:tracer(Nodename, Type, Data): An independent tracer is started on the node (Nodename) and the node is added to the list of traced nodes.

Note

dbg:tracer(Nodename, Type, Data) is not equivalent to dbg:n/1. While dbg:n/1 starts a process tracer which redirects all trace information to a process tracer on the local node (that is, the trace control node), dbg:tracer/3 starts any type of tracer, independent of the type of tracer on the trace control node.

Managing nodes

dbg can trace processes and functions on other Erlang nodes in a distributed system.

dbg:cn(Nodename) (Clear Node): Removes Nodename from the list. Tracing already active on that node continues but new global tp/p calls won't affect it.

dbg:ln() (List Nodes): Shows the list of currently traced nodes.

Trace Ports for Lower Overhead

For high-volume tracing, sending messages to an Erlang process can be too slow. A trace port is an Erlang port to a dynamically linked-in driver that handles trace messages directly, without the overhead of sending them as messages to an Erlang process. Using a trace port significantly lowers the overhead imposed by tracing.

Creating Trace Ports (dbg:trace_port/2)

dbg:trace_port(Type, Parameters) returns a fun/0 that, when called, creates and returns a port. This fun is then passed as the second argument to dbg:tracer(port, Fun).

Two trace drivers are available: the file and the ip trace drivers.

file: Writes trace messages to binary file(s).

Parameters: Filename or a wrap files specification: {Filename, wrap, Suffix} {Filename, wrap, Suffix, WrapSize} {Filename, wrap, Suffix, WrapSize, WrapCnt} {Filename, wrap, Suffix, {time, WrapTime}, WrapCnt} Wrap files limit disk space by rotating through WrapCnt files, each up to WrapSize or open for WrapTime.

ip: Opens a TCP/IP listening port. A client connects to receive trace messages.

Parameters: PortNumber or {PortNumber, QueSize}.

The file trace driver expects a filename or a wrap files specification as parameter. A file is written with a high degree of buffering, which is why there is no guarantee that all are saved in the file in case of a system crash.

A wrap files specification is used to limit the disk space consumed by the trace. The trace is written to a limited number of files each with a limited size. The actual filenames are Filename ++ SeqCnt ++ Suffix, where SeqCnt counts as a decimal string from 0 to WrapCnt and then around again from 0. When a trace term written to the current file makes it longer than WrapSize, that file is closed, and if the number of files in this wrap trace is as many as WrapCnt the oldest file is deleted, and a new file is opened to become the current. Thus, when a wrap trace has been stopped, there are at most WrapCnt trace files saved with a size of at least WrapSize (but not much larger), except for the last file that might even be empty. The default values are WrapSize = 128*1024 and WrapCnt = 8.

The SeqCnt values in the filenames are all in the range 0 through WrapCnt with a gap in the circular sequence. The gap is needed to find the end of the trace.

If the WrapSize is specified as {time, WrapTime}, the current file is closed when it has been open more than WrapTime milliseconds, regardless of it being empty or not.

The ip trace driver has a queue of QueSize messages waiting to be delivered. If the driver cannot deliver messages as fast as they are produced by the runtime system, a special message is sent, which indicates how many messages that are dropped. That message will arrive at the handler function specified in dbg:trace_client/3 as the tuple {drop, N} where N is the number of consecutive messages dropped. In case of heavy tracing, drops are likely to occur, and they surely occur if no client is reading the trace messages. The default value of QueSize is 200.

Reading Trace Port Data (dbg:trace_client/2,3)

dbg:trace_client(Type, Parameters) Starts a trace client that reads the output created by a trace port driver (see dbg:trace_port/2) and handles it in mostly the same way as a tracer process created by the dbg:tracer/0 function. dbg:trace_client(Type, Parameters, HandlerSpec) This function works exactly as dbg:trace_client/2, but allows you to write your own handler function.

If Type is file, the client reads all trace messages stored in the file named Filename or specified by WrapFilesSpec (must be the same as used when creating the trace) and lets the default handler function format the messages on the console. This is one way to interpret the data stored in a file by the file trace port driver.

If Type is follow_file, the client behaves as in the file case, but keeps trying to read (and process) more data from the file until stopped by dbg:stop_trace_client/1. WrapFilesSpec is not allowed as second argument for this Type.

If Type is ip, the client connects to the TCP/IP port PortNumber on the host Hostname, from where it reads trace messages until the TCP/IP connection is closed. If no Hostname is specified, the local host is assumed.

The handler function works mostly as the one described in dbg:tracer/2, but must also be prepared to handle trace messages of the form {drop, N}, where N is the number of dropped messages. This pseudo trace message will only occur if the ip trace driver is used.

For trace type file, the pseudo trace message end_of_trace will appear at the end of the trace. The return value from the handler function is in this case ignored.

Example: Using an IP trace port and connecting to it from another node As an example, one can let trace messages be sent over the network to another Erlang node (preferably not distributed), where the formatting occurs.

On the node stack there exists an Erlang node ant@stack. In the shell, type the following:

ant@stack> dbg:tracer(port, dbg:trace_port(ip, 4711)).
<0.17.0>
ant@stack> dbg:p(self(), send).
{ok,1}

All trace messages are now sent to the trace port driver, which in turn listens for connections on the TCP/IP port 4711. If we want to see the messages on another node, preferably on another host, we do like this:

1> dbg:trace_client(ip, {"stack", 4711}).
<0.42.0>

If we now send a message from the shell on the node ant@stack, where all sends from the shell are traced:

ant@stack> self() ! hello.
hello

The following will appear at the console on the node that started the trace client:

(<0.23.0>) <0.23.0> ! hello
(<0.23.0>) <0.22.0> ! {shell_rep,<0.23.0>,{value,hello,[],[]}}

The last line is generated due to internal message passing in the Erlang shell. The pids will vary.

Controlling Trace Ports

dbg:flush_trace_port() / dbg:flush_trace_port(Node): Flushes internal buffers of the trace port driver on the local/specified node (currently for file driver).

dbg:trace_port_control(Operation) / dbg:trace_port_control(Node, Operation):

Operation = flush: Same as above.

Operation = get_listen_port: For ip driver, returns {ok, IpPortNumber}.

dbg:stop_trace_client(Pid): Shuts down the trace client Pid.

Sequential Tracing

The dbg module is primarily targeted towards tracing through the trace:process/4 function. It is sometimes desired to trace messages in a more delicate way, which can be done with the help of the seq_trace module.

seq_trace implements sequential tracing (known in the AXE10 world, and sometimes called "forlopp tracing"). dbg can interpret messages generated from seq_trace and the same tracer function for both types of tracing can be used. The seq_trace messages can also be sent to a trace port for further analysis.

As a match specification can turn on sequential tracing, the combination of dbg and seq_trace can be powerful. This brief example shows a session where sequential tracing is used to trace the dbg module and the trace itself:

1> dbg:tracer().
{ok,<0.30.0>}
2> {ok, Tracer} = dbg:get_tracer().
{ok,<0.31.0>}
3> seq_trace:set_system_tracer(Tracer).
false
4> dbg:tp(dbg, get_tracer, 0, [{[],[],[{set_seq_token, send, true}]}]).
{ok,[{matched,nonode@nohost,1},{saved,1}]}
5> dbg:p(all,call).
{ok,[{matched,nonode@nohost,22}]}
6> dbg:get_tracer(), seq_trace:set_token([]).
(<0.25.0>) call dbg:get_tracer()
SeqTrace [0]: (<0.25.0>) <0.30.0> ! {<0.25.0>,get_tracer} [Serial: {2,4}]
SeqTrace [0]: (<0.30.0>) <0.25.0> ! {dbg,{ok,<0.31.0>}} [Serial: {4,5}]
{1,0,5,<0.30.0>,4}

This session sets the system_tracer to the same process as the ordinary tracer process (i. e. <0.31.0>) and sets the trace pattern for the function dbg:get_tracer to one that has the action of setting a sequential token. When the function is called by a traced process (all processes are traced in this case), the process gets "contaminated" by the token and seq_trace messages are sent both for the server request and the response. The seq_trace:set_token([]) after the call clears the seq_trace token, which is why no messages are sent when the answer propagates via the shell to the console port. Otherwise the output would have been more noisy.

Avoiding Overloads

Tracing can generate a significant amount of data, potentially overwhelming your system if not managed carefully. To prevent performance degradation or even crashes, consider these strategies:

Time-Limited Tracing: One effective method is to automatically stop tracing after a set period.

dbg:tracer(), dbg:p(all,[c]), dbg:tpl(lists,map,x), timer:sleep(1000), dbg:stop().

Be Specific:

Processes: Instead of dbg:p(all, Flags)., try to pinpoint specific processes if you know which ones are relevant: dbg:p(Pid, Flags). You can also trace newly spawned processes with dbg:p(new, Flags)..

Modules & Functions: Rather than tracing all calls, narrow down to specific modules and functions with dbg:tp/2 or dbg:tpl/2

Use Match Specifications: For example, to only trace calls to my_module:my_function/1 when the first argument is the atom error:

dbg:tpl(my_module, my_function, dbg:fun2ms(fun([error])->true end)).

Limit Trace Flags: Only enable the flags essential for your debugging task (e.g., m for message passing, c for function calls).

Trace to File for High Volumes: If you anticipate a large volume of trace data, tracing directly to the console can become a bottleneck. Consider tracing to a file instead, see trace ports.

dbg:tracer(port, {file, "trace_output.log"}),
% ... your other dbg commands ...
timer:sleep(5000),
dbg:stop().

Avoiding Deadlocks

When tracing function calls on a group leader process (an I/O process), there is risk of causing a deadlock. This will happen if a group leader process generates a trace message and the tracer process, by calling the trace handler function, sends an I/O request to the same group leader. The problem can only occur if the trace handler prints to the tty using an io function such as format/2. Note that when dbg:p(all, call) is called, IO processes are also traced. Here is an example:

%% Using a default line editing shell
1> dbg:tracer(process, {fun(Msg,_) -> io:format("~p~n", [Msg]), 0 end, 0}).
{ok,<0.37.0>}
2> dbg:p(all, [call]).
{ok,[{matched,nonode@nohost,25}]}
3> dbg:tp(mymod,[{'_',[],[]}]).
{ok,[{matched,nonode@nohost,0},{saved,1}]}
4> mymod: % TAB pressed here
%% -- Deadlock --

Here is another example:

%% Using a shell without line editing (oldshell)
1> dbg:tracer(process).
{ok,<0.31.0>}
2> dbg:p(all, [call]).
{ok,[{matched,nonode@nohost,25}]}
3> dbg:tp(lists,[{'_',[],[]}]).
{ok,[{matched,nonode@nohost,0},{saved,1}]}
% -- Deadlock --

The reason we get a deadlock in the first example is because when TAB is pressed to expand the function name, the group leader (which handles character input) calls mymod:module_info(). This generates a trace message which, in turn, causes the tracer process to send an IO request to the group leader (by calling io:format/2). We end up in a deadlock.

In the second example we use the default trace handler function. This handler prints to the tty by sending IO requests to the user process. When Erlang is started in the oldshell mode, the shell process will have user as its group leader and so will the tracer process in this example. Since user calls functions in lists we end up in a deadlock as soon as the first IO request is sent.

Here are a few suggestions for avoiding deadlock:

Do not trace the group leader of the tracer process. If tracing has been switched on for all processes, call dbg:p(TracerGLPid, clear) to stop tracing the group leader (TracerGLPid). process_info(TracerPid, group_leader) tells you which process this is (TracerPid is returned from dbg:get_tracer/0).

Do not trace the user process if using the default trace handler function.

In your own trace handler function, call erlang:display/1 instead of an io function or, if user is not used as group leader, print to user instead of the default group leader. Example: io:format(user, Str, Args).

Getting Information and Help

dbg:i() (Information): Displays information about all currently traced processes and ports and their active trace flags.

dbg:h() (Help): Lists available help items.

dbg:h(Item :: atom()): Gives brief help for a specific dbg function or concept (e.g., dbg:h(tp)).

dbg:get_tracer() / dbg:get_tracer(Node): Returns the process, port, or tracer module handling traces on the local/specified node.

Consult the dbg module documentation.

dbg

The Text Based Trace Facility

This module implements a text based interface to the trace:process/4, trace:port/4, and trace:function/4 BIFs, simplifying tracing of functions, processes, ports, and messages.

The Tracing in Erlang with dbg Users guide explains how to quickly get started on tracing function calls, complex systems and more.

Summary

Types

built_in_alias()

handler_spec()

match_desc()

match_info()

match_num()

match_pattern()

match_spec()

session()

A dbg session that can be used by session/2 to create isolated debugging sessions.

tp_arity()

tp_function()

tp_id()

tp_match_spec()

tp_module()

trace_wrap_file_size()

trace_wrap_files_spec()

Functions

c(Mod, Fun, Args)

Equivalent to c(Mod, Fun, Args, all).

c(Mod, Fun, Args, Flags)

Evaluates the expression apply(Mod, Fun, Args) with the trace flags in Flags set.

cn(Nodename)

Clears a node from the list of traced nodes.

ctp()

Equivalent to ctp({'_', '_', '_'}).

ctp(ModuleOrMFA)

Disables call tracing for one or more functions specified by ModuleOrMFA.

ctp(Module, Function)

Equivalent to ctp({Module, Function, '_'}).

ctp(Module, Function, Arity)

Equivalent to ctp({Module, Function, Arity}).

ctpe(Event)

Clears match specifications for the specified trace event (send or 'receive'), reverting to the default of tracing all triggered events.

ctpg()

Equivalent to ctpg({'_', '_', '_'}).

ctpg(ModuleOrMFA)

Disables global call tracing for one or more functions specified by ModuleOrMFA.

ctpg(Module, Function)

Equivalent to ctpg({Module, Function, '_'}).

ctpg(Module, Function, Arity)

Equivalent to ctpg({Module, Function, Arity}).

ctpl()

Equivalent to ctpl({'_', '_', '_'}).

ctpl(ModuleOrMFA)

Disables local call tracing for one or more functions specified by ModuleOrMFA.

ctpl(Module, Function)

Equivalent to ctpl({Module, Function, '_'}).

ctpl(Module, Function, Arity)

Equivalent to ctpl({Module, Function, Arity}).

dtp()

Forgets all match specifications saved during calls to tp/2.

dtp(N)

Forgets a specific match specification saved during calls to tp/2.

flush_trace_port()

Equivalent to flush_trace_port(node()).

flush_trace_port(Nodename)

Equivalent to trace_port_control(Nodename, flush).

fun2ms(LiteralFun)

Pseudo function that by means of a parse transform translates the literal fun typed as parameter in the function call to a match specification.

get_tracer()

Equivalent to get_tracer(node()).

get_tracer(Nodename)

Returns the process, port, or tracer module to which all trace messages are sent.

h()

Gives a list of items for brief online help.

h(Item)

Gives a brief help text for functions in the dbg module.

i()

Displays information about all traced processes and ports.

ln()

Shows the list of traced nodes on the console.

ltp()

Lists all match specifications previously used in the session.

n(Nodename)

Adds a remote node (Nodename) to the list of nodes where tracing is performed.

p(Item)

Equivalent to p(Item, [m]).

p(Item, Flags)

Traces Item in accordance to the value specified by Flags.

rtp(Name)

Reads match specifications from a text file (possibly) generated by the wtp/1 function.

session(Session, Fun)

Runs dbg commands using the provides session, or creates a session for the duration of the call if a session name is provided.

session_create(Name)

Create a new dbg session with the given Name.

session_destroy(Session)

Destroys a dbg session/0.

stop()

Stops the dbg server, clears all trace flags for all processes, clears all trace patterns for all functions, clears trace patterns for send/receive, shuts down all trace clients, and closes all trace ports.

stop_trace_client(Pid)

Shuts down a previously started trace client.

tp(ModuleOrMFA, MatchSpec)

Enables call trace for one or more exported functions specified by ModuleOrMFA.

tp(Module, Function, MatchSpec)

Equivalent to tp({Module, Function, '_'}, MatchSpec).

tp(Module, Function, Arity, MatchSpec)

Equivalent to tp({Module, Function, Arity}, MatchSpec).

tpe(Event, MatchSpec)

Associates a match specification with trace event send or 'receive'.

tpl({Module, Function, Arity}, MatchSpec)

Enables call trace for one or more functions specified by ModuleOrMFA.

tpl(Module, Function, MatchSpec)

Equivalent to tpl({Module, Function, '_'}, MatchSpec).

tpl(Module, Function, Arity, MatchSpec)

Equivalent to tpl({Module, Function, Arity}, MatchSpec).

trace_client(Type, Parameters)

Starts a trace client that reads the output created by a trace port driver (see trace_port/2) and handles it in mostly the same way as a tracer process created by the tracer/0 function.

trace_client(Type, Parameters, HandlerSpec)

This function works exactly as trace_client/2, but allows you to write your own handler function.

trace_port(Type, Parameters)

Creates a trace-port-generating fun that is suitable as the second argument to tracer/2.

trace_port_control(Operation)

Equivalent to trace_port_control(node(), Operation).

trace_port_control(Nodename, Operation)

This function is used to do a control operation on the active trace port driver on the given node (Nodename).

tracer()

Starts a server on the local node that will be the recipient of all trace messages.

tracer(Type, Data)

Starts a tracer server with additional parameters on the local node.

tracer(Nodename, Type, Data)

This function is equivalent to tracer/2, but acts on the given node.

wtp(Name)

Saves all match specifications saved during the session (by calls to tp/2 or tpl/2), as well as built-in match specifications, in a text file with the name designated by Name.

Types

built_in_alias()

(not exported)

-type built_in_alias() :: x | c | cx.

handler_spec()

(not exported)

-type handler_spec() ::
          {HandlerFun :: fun((Event :: term(), Data :: term()) -> NewData :: term()),
           InitialData :: term()}.

match_desc()

(not exported)

-type match_desc() :: [match_info()].

match_info()

(not exported)

-type match_info() :: {saved, tp_id()} | match_num().

match_num()

(not exported)

-type match_num() :: {matched, node(), integer()} | {matched, node(), 0, RPCError :: term()}.

match_pattern()

(not exported)

-type match_pattern() :: atom() | list().

match_spec()

-type match_spec() :: [{match_pattern(), [_], [_]}].

session()

-opaque session()

A dbg session that can be used by session/2 to create isolated debugging sessions.

tp_arity()

(not exported)

-type tp_arity() :: arity() | '_'.

tp_function()

(not exported)

-type tp_function() :: atom() | '_'.

tp_id()

(not exported)

-type tp_id() :: pos_integer().

tp_match_spec()

(not exported)

-type tp_match_spec() :: tp_id() | built_in_alias() | [] | match_spec().

tp_module()

(not exported)

-type tp_module() :: module() | '_'.

trace_wrap_file_size()

(not exported)

-type trace_wrap_file_size() :: non_neg_integer() | {time, WrapTime :: pos_integer()}.

trace_wrap_files_spec()

(not exported)

-type trace_wrap_files_spec() ::
          {file:name_all(), wrap, Suffix :: string()} |
          {file:name_all(), wrap, Suffix :: string(), WrapSize :: trace_wrap_file_size()} |
          {file:name_all(),
           wrap,
           Suffix :: string(),
           WrapSize :: trace_wrap_file_size(),
           WrapCnt :: pos_integer()}.

Functions

c(Mod, Fun, Args)

-spec c(Mod :: module(), Fun :: atom(), Args :: [term()]) -> term().

Equivalent to c(Mod, Fun, Args, all).

c(Mod, Fun, Args, Flags)

-spec c(Mod :: module(), Fun :: atom(), Args :: [term()], Flags :: term()) -> term().

Evaluates the expression apply(Mod, Fun, Args) with the trace flags in Flags set.

c stands for call.

This is a convenient way to trace processes from the Erlang shell.

cn(Nodename)

-spec cn(Nodename) -> ok when Nodename :: node().

Clears a node from the list of traced nodes.

cn stands for clear node.

Subsequent calls to tp/2 and p/2 will not consider that node, but tracing already activated on the node will continue to be in effect.

Returns ok. This call cannot fail.

ctp()

-spec ctp() -> {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctp({'_', '_', '_'}).

ctp(ModuleOrMFA)

-spec ctp(Module | {Module, Function, Arity}) -> {ok, MatchDesc :: match_desc()} | {error, term()}
             when Module :: tp_module(), Function :: tp_function(), Arity :: tp_arity().

Disables call tracing for one or more functions specified by ModuleOrMFA.

If ModuleOrMFA is an atom (a module name), this function call is equivalent to ctp({ModuleOrMFA, '_', '_'}).

Otherwise, ModuleOrMFA should be {Module, Function, Arity}.

ctp stands for clear trace pattern.

The semantics of ModuleOrMFA is the same as for the corresponding function specification in tp/2 or tpl/2. Both local and global call trace is disabled.

The return value reflects how many functions that matched, and is constructed as described in tp/2, except that no {saved, N} tuple is returned.

ctp(Module, Function)

-spec ctp(Module :: tp_module(), Function :: tp_function()) ->
             {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctp({Module, Function, '_'}).

ctp(Module, Function, Arity)

-spec ctp(Module :: tp_module(), Function :: tp_function(), Arity :: tp_arity()) ->
             {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctp({Module, Function, Arity}).

ctpe(Event)

(since OTP 19.0)

-spec ctpe(Event) -> {ok, MatchDesc} | {error, term()}
              when
                  Event :: send | 'receive',
                  MatchDesc :: [MatchNum],
                  MatchNum :: {matched, node(), 1} | {matched, node(), 0, RPCError :: term()}.

Clears match specifications for the specified trace event (send or 'receive'), reverting to the default of tracing all triggered events.

ctpe stands for clear trace pattern event.

ctpg()

-spec ctpg() -> {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpg({'_', '_', '_'}).

ctpg(ModuleOrMFA)

-spec ctpg(Module | {Module, Function :: tp_function(), Arity :: tp_arity()}) ->
              {ok, MatchDesc :: term()} | {error, term()}
              when Module :: tp_module().

Disables global call tracing for one or more functions specified by ModuleOrMFA.

If ModuleOrMFA is an atom (a module name), this function call is equivalent to ctpg({ModuleOrMFA, '_', '_'}).

Otherwise, ModuleOrMFA should be {Module, Function, Arity}.

ctpg stands for clear trace pattern global.

This function works as ctp/1, but only disables tracing set up with tp/2 (not with tpl/2).

ctpg(Module, Function)

-spec ctpg(Module :: tp_module(), Function :: tp_function()) ->
              {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpg({Module, Function, '_'}).

ctpg(Module, Function, Arity)

-spec ctpg(Module :: tp_module(), Function :: tp_function(), Arity :: tp_arity()) ->
              {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpg({Module, Function, Arity}).

ctpl()

-spec ctpl() -> {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpl({'_', '_', '_'}).

ctpl(ModuleOrMFA)

-spec ctpl(Module | {Module, Function :: tp_function(), Arity :: tp_arity()}) ->
              {ok, MatchDesc :: term()} | {error, term()}
              when Module :: tp_module().

Disables local call tracing for one or more functions specified by ModuleOrMFA.

If ModuleOrMFA is an atom (a module name), this function call is equivalent to ctpl({ModuleOrMFA, '_', '_'}).

Otherwise, ModuleOrMFA should be {Module, Function, Arity}.

ctpl stands for clear trace pattern local.

This function works as ctp/1, but only disables tracing set up with tpl/2 (not with tp/2).

ctpl(Module, Function)

-spec ctpl(Module :: tp_module(), Function :: tp_function()) ->
              {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpl({Module, Function, '_'}).

ctpl(Module, Function, Arity)

-spec ctpl(Module :: tp_module(), Function :: tp_function(), Arity :: tp_arity()) ->
              {ok, MatchDesc :: match_desc()} | {error, term()}.

Equivalent to ctpl({Module, Function, Arity}).

dtp()

-spec dtp() -> ok.

Forgets all match specifications saved during calls to tp/2.

dtp stands for delete trace patterns.

Removing all saved match specifications is useful before restoring other match specifications from a file with rtp/1. Use dtp/1 to delete specific saved match specifications.

dtp(N)

-spec dtp(N) -> ok when N :: tp_id().

Forgets a specific match specification saved during calls to tp/2.

dtp stands for delete trace pattern.

flush_trace_port()

-spec flush_trace_port() -> term().

Equivalent to flush_trace_port(node()).

flush_trace_port(Nodename)

-spec flush_trace_port(Nodename :: node()) -> ok | {error, Reason :: term()}.

Equivalent to trace_port_control(Nodename, flush).

fun2ms(LiteralFun)

-spec fun2ms(LiteralFun) -> MatchSpec
                when LiteralFun :: fun((term()) -> term()), MatchSpec :: match_spec().

Pseudo function that by means of a parse transform translates the literal fun typed as parameter in the function call to a match specification.

The meaning of "literal" is that the fun needs to textually be written as the argument of the function call; it cannot be held in a variable which in turn is passed to the function. Furthermore, the parse transform module ms_transform must be enabled. The easiest way to enable it is by adding the following line to the source file:

-include_lib("stdlib/include/ms_transform.hrl").

Failing to include ms_transform.hrl in the source will result in a runtime error, not a compile-time error.

This function can also be invoked directly from the Erlang shell, as shown in the examples that follow.

The head of the fun must be a single pattern that matches a list. That pattern will be used to match the arguments for the call:

Examples:

1> dbg:fun2ms(fun([_,_]) -> true end).
[{['_','_'],[],[true]}]
2> dbg:fun2ms(fun(Args) when length(Args) > 6 -> true end).
[{'$1',[{'>',{length,'$1'},6}],[true]}]

The first match specification matches when a function having two arguments is called. The second matches when a function with more than 6 arguments is called.

Examples:

1> dbg:fun2ms(fun(42) -> true end).
Error: dbg:fun2ms requires fun with single variable or list parameter
{error,transform_error}
2> dbg:fun2ms(fun([<<H,T/binary>>]) -> true end).
Error: fun head contains bit syntax matching of variable 'H', which cannot be translated into match_spec
{error,transform_error}

The preceding two examples show what happens when a fun cannot be translated into a match specification. In the first example, the fun head connot possibly match a list. In the second example, an attempt is made to take apart a binary using the bit syntax, which is currently not supported in match specifications.

However, note that literal binaries can be matched:

1> dbg:fun2ms(fun([<<"abc">>]) -> true end).
[{[<<"abc">>],[],[true]}]

Match specifications support a large subset of the guard expressions supported by Erlang, but not all. For example, updating a map is currently not supported:

1> dbg:fun2ms(fun([M]) when map_size(M#{a => b}) > 2 -> true end).
Error: the language element map (in guard) cannot be translated into match_spec
{error,transform_error}

However, creating a map in a guard is allowed:

1> dbg:fun2ms(fun([M]) when map_size(#{a => b}) > 2 -> true end).
[{['$1'],[{'>',{map_size,#{a => b}},2}],[true]}]

Variables from the environment can be imported, so this works:

1> X = 3.
3
2> dbg:fun2ms(fun([M,N]) when N > X  -> return_trace() end).
[{['$1','$2'],[{'>','$2',{const,3}}],[{return_trace}]}]

The imported variables will be replaced by const expressions, which is consistent with the static scoping for Erlang funs.

In the body of the fun, only guard expressions and calls to the special functions for tracing are allowed.

Examples:

1> dbg:fun2ms(fun([A]) when is_atom(A) -> return_trace() end).
[{['$1'],[{is_atom,'$1'}],[{return_trace}]}]
2> dbg:fun2ms(fun(_) -> erlang:garbage_collect() end).
Error: fun containing the remote function call 'erlang:garbage_collect/0' (called in body) cannot be translated into match_spec
{error,transform_error}

Warning

If the parse transform is not applied to a module which calls dbg:fun2ms/1, the call will fail in runtime with a badarg exception.

More information is available in the documentation for module ms_transform in STDLIB.

get_tracer()

-spec get_tracer() -> term().

Equivalent to get_tracer(node()).

get_tracer(Nodename)

-spec get_tracer(Nodename) -> {ok, Tracer}
                    when Nodename :: atom(), Tracer :: port() | pid() | {module(), term()}.

Returns the process, port, or tracer module to which all trace messages are sent.

h()

-spec h() -> ok.

Gives a list of items for brief online help.

h stands for help.

h(Item)

-spec h(Item) -> ok when Item :: atom().

Gives a brief help text for functions in the dbg module.

h stands for help.

The available items can be listed by calling dbg:h/0.

i()

-spec i() -> ok.

Displays information about all traced processes and ports.

i stands for information.

ln()

-spec ln() -> ok.

Shows the list of traced nodes on the console.

ln stands for list nodes.

ltp()

-spec ltp() -> ok.

Lists all match specifications previously used in the session.

ltp stands for list trace patterns.

This function lists all match specifications previously saved during calls to tp/2 and tpl/2, as well as all built-in match specifications. This avoids having to re-type complicated match specifications. Note that the match specifications are lost if stop/0 is called.

Match specifications can be saved in a file (if a read-write file system is present) for use in later debugging sessions; see wtp/1 and rtp/1.

There are three built-in trace patterns:

exception_trace, x - sets a trace which will show function names, parameters, return values, and exceptions raised from functions

caller_trace, c - sets a trace that displays function names, parameters, and information about which function called it

caller_exception_trace, cx - combines exception_trace and caller_trace

Here is an example that shows how to use a built-in match specification:

1> dbg:tracer().
{ok,<0.90.0>}
2> dbg:tp(lists, seq, 2, cx).
{ok,[{matched,nonode@nohost,1},{saved,cx}]}
3> dbg:p(self(), call).
{ok,[{matched,nonode@nohost,1}]}
4> lists:seq(1, 5).
(<0.88.0>) call lists:seq(1,5) ({erl_eval,do_apply,7,{"erl_eval.erl",904}})
[1,2,3,4,5]
(<0.88.0>) returned from lists:seq/2 -> [1,2,3,4,5]

n(Nodename)

-spec n(Nodename) -> {ok, Nodename} | {error, Reason} when Nodename :: node(), Reason :: term().

Adds a remote node (Nodename) to the list of nodes where tracing is performed.

n stands for node.

The dbg server keeps a list of nodes where tracing should be performed. Whenever a tp/2 call or a p/2 call is made, it is executed for all nodes in this list including the local node (except for p/2 with a specific pid/0 or port/0 as first argument, in which case the command is executed only on the node where the designated process or port resides).

When this function is called, it starts a tracer process on the remote node, which will send all trace messages to the tracer process on the local node (via the Erlang distribution). If no tracer process is running on the local node, the error reason no_local_tracer is returned. The tracer process on the local node must be started with the tracer/0,2 function.

If Nodename is the local node, the error reason cant_add_local_node is returned.

If a trace port (see trace_port/2) is running on the local node, remote nodes cannot be traced with a tracer process. The error reason cant_trace_remote_pid_to_local_port is returned. However, a trace port can be started on the remote node with the tracer/3 function.

The function will also return an error if the node Nodename is not reachable.

p(Item)

-spec p(Item :: term()) -> {ok, MatchDesc :: term()} | {error, term()}.

Equivalent to p(Item, [m]).

p(Item, Flags)

-spec p(Item :: term(), Flags :: term()) -> {ok, MatchDesc} | {error, term()}
           when
               MatchDesc :: [MatchNum],
               MatchNum :: {matched, node(), integer()} | {matched, node(), 0, RPCError},
               RPCError :: term().

Traces Item in accordance to the value specified by Flags.

p stands for process.

The following kind of values are allowed for Item:

pid/0 or port/0 - The corresponding process or port is traced. The process or port can be a remote process or port (on another Erlang node). The node must be in the list of traced nodes (see n/1 and tracer/3).

all - All processes and ports in the system as well as all processes and ports created hereafter are to be traced.

processes - All processes in the system as well as all processes created hereafter are to be traced.

ports - All ports in the system as well as all ports created hereafter are to be traced.

new - All processes and ports created after the call are to be traced.

new_processes - All processes created after the call are to be traced.

new_ports - All ports created after the call are to be traced.

existing - All existing processes and ports are traced.

existing_processes - All existing processes are traced.

existing_ports - All existing ports are traced.

atom/0 - The process or port with the corresponding registered name is traced. The process or port can on another Erlang node. The node must be in the list of traced nodes (see n/1 and tracer/3).

integer/0 - The process <0.Item.0> is traced.

{X, Y, Z} - The process <X.Y.Z> is traced.

string/0 - If the Item is a string "<X.Y.Z>" as returned from pid_to_list/1, the process <X.Y.Z> is traced.

When enabling an Item that represents a group of processes, the Item is enabled on all nodes added with the n/1 or tracer/3 function.

Flags can be a single atom or a list of flags. The available flags are:

s (send) - Traces the messages the process or port sends.

r (receive) - Traces the messages the process or port receives.

m (messages) - Traces the messages the process or port receives and sends.

c (call) - Traces global function calls for the process according to the trace patterns set in the system (see tp/2).

p (procs) - Traces process related events to the process.

ports - Traces port related events to the port.

sos (set on spawn) - Lets all processes created by the traced process inherit the trace flags of the traced process.

sol (set on link) - Lets another process, P2, inherit the trace flags of the traced process whenever the traced process links to P2.

sofs (set on first spawn) - This is the same as sos, but only for the first process spawned by the traced process.

sofl (set on first link) - This is the same as sol, but only for the first call to link/1 by the traced process.

all - Sets all flags except silent.

clear - Clears all flags.

The list can also include any of the flags allowed in trace:process/4 and trace:port/4.

This function returns either an error tuple or an {ok, List} tuple. The List consists of specifications of how many processes and ports that matched (in the case of a single pid exactly 1). The specification of matched processes is {matched, Node, N}. If the remote processor call (using rpc) to a remote node fails, the rpc error message is returned as the fourth element in the tuple and the number of matched processes is 0.

rtp(Name)

-spec rtp(Name) -> ok | {error, Error} when Name :: string(), Error :: term().

Reads match specifications from a text file (possibly) generated by the wtp/1 function.

rtp stands for read trace patterns.

The function verifies that the syntax of all match specifications are correct. If any error in any match specification is found, none of the match specifications are added to the list of saved match specifications for the running system.

The match specifications in the file are merged with the current match specifications, so that no duplicates are generated. Use ltp/0 to see what numbers were assigned to the specifications from the file.

The function will return an error tuple, either due to I/O problems (like a non-existing or non-readable file) or due to file format problems. In the latter case, Reason is in a more or less textual format, giving a hint to what is causing the problem.

session(Session, Fun)

(since OTP 27.0)

-spec session(atom(), fun(() -> term())) -> term();
             (session(), fun(() -> term())) -> term().

Runs dbg commands using the provides session, or creates a session for the duration of the call if a session name is provided.

Any dbg function that is called with in the provided fun will use the session/0 provided instead of the default dbg session. This means that the tracing will be isolated from other tracing users on the system.

The function returns the term that the fun returns.

Example:

1> S = dbg:session_create(my_session).
<0.91.0>
2> dbg:session(S, fun() -> dbg:tracer(), dbg:p(all,c), dbg:tp(lists,seq,x) end).
{ok,[{matched,nonode@nohost,2},{saved,x}]}
3> lists:seq(1, 10).
(<0.89.0>) call lists:seq(1,10)
(<0.89.0>) returned from lists:seq/2 -> [1,2,3,4,5,6,7,8,9,10]
[1,2,3,4,5,6,7,8,9,10]
4> dbg:session_destroy(S).
ok

The state of the session/0 is preserved in between session/2 calls, so you can call session/2 multiple when debugging you application.

Example:

1> S = dbg:session_create(my_session).
<0.91.0>
%% Setup the initial traces
2> dbg:session(S, fun() -> dbg:tracer(), dbg:p(self(),c), dbg:tp(lists,seq,x) end).
{ok,[{matched,nonode@nohost,2},{saved,x}]}
3> lists:seq(1, 3).
(<0.89.0>) call lists:seq(1,3)
(<0.89.0>) returned from lists:seq/2 -> [1,2,3]
[1,2,3]
%% Add an additional trace pattern
4> dbg:session(S, fun() -> dbg:tpl(lists,seq_loop,x) end).
ok
5> lists:seq(1, 3).
(<0.89.0>) call lists:seq(1,3)
(<0.89.0>) call lists:seq_loop(3,3,[])
(<0.89.0>) call lists:seq_loop(1,1,[2,3])
(<0.89.0>) returned from lists:seq_loop/3 -> [1,2,3]
(<0.89.0>) returned from lists:seq_loop/3 -> [1,2,3]
(<0.89.0>) returned from lists:seq/2 -> [1,2,3]
[1,2,3]
6> dbg:session_destroy(S).
ok

Note

The session functionality is experimental in Erlang/OTP 27 and may change in future releases without notice.

session_create(Name)

(since OTP 27.0)

-spec session_create(atom()) -> session().

Create a new dbg session with the given Name.

The session is linked with the calling process and will be

Multiple sessions can have the same name.

Note

The session functionality is experimental in Erlang/OTP 27 and may change in future releases without notice.

session_destroy(Session)

(since OTP 27.0)

-spec session_destroy(Session :: session()) -> ok.

Destroys a dbg session/0.

This will terminate all started processes and destroy the trace:session/0.

stop()

-spec stop() -> ok.

Stops the dbg server, clears all trace flags for all processes, clears all trace patterns for all functions, clears trace patterns for send/receive, shuts down all trace clients, and closes all trace ports.

stop_trace_client(Pid)

-spec stop_trace_client(Pid) -> ok when Pid :: pid().

Shuts down a previously started trace client.

The Pid argument is the process id returned from the trace_client/2 or trace_client/3 call.

tp(ModuleOrMFA, MatchSpec)

-spec tp(Module | {Module, Function, Arity}, MatchSpec) -> {ok, match_desc()} | {error, term()}
            when
                Module :: tp_module(),
                Function :: tp_function(),
                Arity :: tp_arity(),
                MatchSpec :: tp_match_spec().

Enables call trace for one or more exported functions specified by ModuleOrMFA.

If ModuleOrMFA is an atom (a module name), this function call is equivalent to tp({ModuleOrMFA, '_', '_'}, MatchSpec).

Otherwise, ModuleOrMFA should be {Module, Function, Arity}.

tp stands for trace pattern.

All exported functions matching the {Module, Function, Arity} argument will be concerned, but the match specification may further narrow down the set of function calls generating trace messages.

For a description of the format for the MatchSpec argument, see Match Specifications in Erlang, which explains the general match specification language. The most common generic match specifications used can be found as built-in aliases; see ltp/0 below for details.

The Module, Function and/or Arity parts of the tuple may be specified as the atom '_' which is a wildcard matching all modules, functions, or arities. Note that if the Module is specified as '_', the Function and Arity parts must be specified as '_' as well. The same holds for the Function in relation to Arity.

All nodes added with n/1 or tracer/3 will be affected by this call, and if Module is not '_' the module will be loaded on all nodes.

The function returns either an error tuple or an {ok, List} tuple. The List consists of specifications of how many functions that matched, in the same way as the processes and ports are presented in the return value of p/2.

There may be a tuple {saved, N} in the return value, if the MatchSpec is not []. The integer N can then be used in subsequent calls to this function and will stand as an "alias" for the given expression.

If the match specification is invalid, an {error, Errors} tuple is returned. Errors is as a list of tuples {error, string()}, where the string is a textual explanation of the compilation error. For example:

1> dbg:tp({dbg,ltp,0},[{[],[],[{message, two, arguments}, {noexist}]}]).
{error,
 [{error,"Special form 'message' called with wrong number of
          arguments in {message,two,arguments}."},
  {error,"Function noexist/1 does_not_exist."}]}

tp(Module, Function, MatchSpec)

-spec tp(Module :: tp_module(), Function :: tp_function(), MatchSpec :: tp_match_spec()) ->
            {ok, match_desc()} | {error, term()}.

Equivalent to tp({Module, Function, '_'}, MatchSpec).

tp(Module, Function, Arity, MatchSpec)

-spec tp(Module :: tp_module(),
         Function :: tp_function(),
         Arity :: tp_arity(),
         MatchSpec :: tp_match_spec()) ->
            {ok, match_desc()} | {error, term()}.

Equivalent to tp({Module, Function, Arity}, MatchSpec).

tpe(Event, MatchSpec)

(since OTP 19.0)

-spec tpe(Event, MatchSpec) -> {ok, MatchDesc :: match_desc()} | {error, term()}
             when Event :: send | 'receive', MatchSpec :: tp_match_spec().

Associates a match specification with trace event send or 'receive'.

tpe stands for trace pattern event.

By default all executed send and 'receive' events are traced if enabled for a process. A match specification can be used to filter traced events based on sender, receiver, and/or message content.

For a description of the format for the MatchSpec argument, see Match Specifications in Erlang, which explains the general match specification language.

For send, the matching is done on the list [Receiver, Msg]. Receiver is the process or port identity of the receiver and Msg is the message term. The pid of the sending process can be accessed with the guard function self/0.

For 'receive', the matching is done on the list [Node, Sender, Msg]. Node is the node name of the sender. Sender is the process or port identity of the sender, or the atom undefined if the sender is not known (which may be the case for remote senders). Msg is the message term. The pid of the receiving process can be accessed by calling self/0.

All nodes added with n/1 or tracer/3 will be affected by this call.

The return value is the same as for tp/2. The number of matched events is always 1 as tpe/2 does not accept any form of wildcards for argument Event.

tpl({Module, Function, Arity}, MatchSpec)

-spec tpl(Module | {Module, Function :: tp_function(), Arity :: tp_arity()},
          MatchSpec :: tp_match_spec()) ->
             {ok, MatchDesc :: term()} | {error, term()}
             when Module :: tp_module().

Enables call trace for one or more functions specified by ModuleOrMFA.

If ModuleOrMFA is an atom (a module name), this function call is equivalent to tpl({ModuleOrMFA, '_', '_'}, MatchSpec).

Otherwise, ModuleOrMFA should be {Module, Function, Arity}.

tpl stands for trace pattern local.

This function works as tp/2, but enables tracing for local or remote calls to both local and exported functions.

tpl(Module, Function, MatchSpec)

-spec tpl(Module :: tp_module(), Function :: tp_function(), MatchSpec :: tp_match_spec()) ->
             {ok, match_desc()} | {error, term()}.

Equivalent to tpl({Module, Function, '_'}, MatchSpec).

tpl(Module, Function, Arity, MatchSpec)

-spec tpl(Module :: tp_module(),
          Function :: tp_function(),
          Arity :: tp_arity(),
          MatchSpec :: tp_match_spec()) ->
             {ok, match_desc()} | {error, term()}.

Equivalent to tpl({Module, Function, Arity}, MatchSpec).

trace_client(Type, Parameters)

-spec trace_client(ip, IPClientPortSpec) -> pid()
                      when
                          IPClientPortSpec :: PortNumber | {Hostname, PortNumber},
                          PortNumber :: integer(),
                          Hostname :: string();
                  (Type, Parameters) -> pid()
                      when
                          Type :: file | follow_file,
                          Parameters :: Filename | WrapFilesSpec,
                          Filename :: file:name_all(),
                          WrapFilesSpec :: trace_wrap_files_spec().

Starts a trace client that reads the output created by a trace port driver (see trace_port/2) and handles it in mostly the same way as a tracer process created by the tracer/0 function.

If Type is file, the client reads all trace messages stored in the file named Filename or specified by WrapFilesSpec (must be the same as used when creating the trace) and lets the default handler function format the messages on the console. This is one way to interpret the data stored in a file by the file trace port driver.

If Type is follow_file, the client behaves as in the file case, but keeps trying to read (and process) more data from the file until stopped by stop_trace_client/1. WrapFilesSpec is not allowed as second argument for this Type.

If Type is ip, the client connects to the TCP/IP port PortNumber on the host Hostname, from where it reads trace messages until the TCP/IP connection is closed. If no Hostname is specified, the local host is assumed.

As an example, one can let trace messages be sent over the network to another Erlang node (preferably not distributed), where the formatting occurs.

On the node stack there exists an Erlang node ant@stack. In the shell, type the following:

ant@stack> dbg:tracer(port, dbg:trace_port(ip, 4711)).
<0.17.0>
ant@stack> dbg:p(self(), send).
{ok,1}

All trace messages are now sent to the trace port driver, which in turn listens for connections on the TCP/IP port 4711. If we want to see the messages on another node, preferably on another host, we do like this:

1> dbg:trace_client(ip, {"stack", 4711}).
<0.42.0>

If we now send a message from the shell on the node ant@stack, where all sends from the shell are traced:

ant@stack> self() ! hello.
hello

The following will appear at the console on the node that started the trace client:

(<0.23.0>) <0.23.0> ! hello
(<0.23.0>) <0.22.0> ! {shell_rep,<0.23.0>,{value,hello,[],[]}}

The last line is generated due to internal message passing in the Erlang shell. The pids will vary.

trace_client(Type, Parameters, HandlerSpec)

-spec trace_client(ip, IPClientPortSpec, HandlerSpec) -> pid()
                      when
                          IPClientPortSpec :: PortNumber | {Hostname, PortNumber},
                          PortNumber :: integer(),
                          Hostname :: string(),
                          HandlerSpec :: handler_spec();
                  (Type, Parameters, HandlerSpec) -> pid()
                      when
                          Type :: file | follow_file,
                          Parameters :: Filename | WrapFilesSpec,
                          Filename :: string() | [string()] | atom(),
                          WrapFilesSpec :: trace_wrap_files_spec(),
                          HandlerSpec :: handler_spec().

This function works exactly as trace_client/2, but allows you to write your own handler function.

The handler function works mostly as the one described in tracer/2, but must also be prepared to handle trace messages of the form {drop, N}, where N is the number of dropped messages. This pseudo trace message will only occur if the ip trace driver is used.

For trace type file, the pseudo trace message end_of_trace will appear at the end of the trace. The return value from the handler function is in this case ignored.

trace_port(Type, Parameters)

-spec trace_port(ip, IpPortSpec) -> fun(() -> port())
                    when
                        IpPortSpec :: PortNumber | {PortNumber, QueSize},
                        PortNumber :: integer(),
                        QueSize :: integer();
                (file, Parameters) -> fun(() -> port())
                    when
                        Parameters :: Filename | WrapFilesSpec,
                        Filename :: file:name_all(),
                        WrapFilesSpec :: trace_wrap_files_spec().

Creates a trace-port-generating fun that is suitable as the second argument to tracer/2.

Example:

dbg:tracer(port, dbg:trace_port(ip, 4711)).

A trace port is an Erlang port to a dynamically linked-in driver that handles trace messages directly, without the overhead of sending them as messages to an Erlang process. Using a trace port significantly lowers the overhead imposed by tracing.

Two trace drivers are currently implemented: the file and the ip trace drivers.

The file driver sends all trace messages into one or several binary files, from where they later can be fetched and processed with the trace_client/2 function.

The ip driver opens a TCP/IP port listening port. When a client (preferably started by calling trace_client/2 on another Erlang node) connects, all trace messages are sent over the IP network for further processing by the remote client.

The file trace driver expects a filename or a wrap files specification as parameter. A file is written with a high degree of buffering, which is why there is no guarantee that all are saved in the file in case of a system crash.

A wrap files specification is used to limit the disk space consumed by the trace. The trace is written to a limited number of files each with a limited size. The actual filenames are Filename ++ SeqCnt ++ Suffix, where SeqCnt counts as a decimal string from 0 to WrapCnt and then around again from 0. When a trace term written to the current file makes it longer than WrapSize, that file is closed, and if the number of files in this wrap trace is as many as WrapCnt the oldest file is deleted, and a new file is opened to become the current. Thus, when a wrap trace has been stopped, there are at most WrapCnt trace files saved with a size of at least WrapSize (but not much larger), except for the last file that might even be empty. The default values are WrapSize = 128*1024 and WrapCnt = 8.

The SeqCnt values in the filenames are all in the range 0 through WrapCnt with a gap in the circular sequence. The gap is needed to find the end of the trace.

If the WrapSize is specified as {time, WrapTime}, the current file is closed when it has been open more than WrapTime milliseconds, regardless of it being empty or not.

The ip trace driver has a queue of QueSize messages waiting to be delivered. If the driver cannot deliver messages as fast as they are produced by the runtime system, a special message is sent, which indicates how many messages that are dropped. That message will arrive at the handler function specified in trace_client/3 as the tuple {drop, N} where N is the number of consecutive messages dropped. In case of heavy tracing, drops are likely to occur, and they surely occur if no client is reading the trace messages. The default value of QueSize is 200.

trace_port_control(Operation)

-spec trace_port_control(Operation :: term()) -> term().

Equivalent to trace_port_control(node(), Operation).

trace_port_control(Nodename, Operation)

-spec trace_port_control(Nodename :: node(), Operation :: term()) ->
                            ok | {ok, Result :: term()} | {error, Reason :: term()}.

This function is used to do a control operation on the active trace port driver on the given node (Nodename).

Which operations are allowed as well as their return values depend on which trace driver is used.

Returns either ok or {ok, Result} if the operation was successful, or {error, Reason} if the current tracer is a process, or if it is a port not supporting the operation.

The allowed values for Operation are:

flush - This function is used to flush the internal buffers held by a trace port driver. Currently only the file trace driver supports this operation. Returns ok.

get_listen_port - Returns {ok, IpPort} where IpPort is the IP port number used by the driver listen socket. Only the ip trace driver supports this operation.

tracer()

-spec tracer() -> {ok, pid()} | {error, already_started}.

Starts a server on the local node that will be the recipient of all trace messages.

All subsequent calls to p/2 will result in messages sent to the newly started trace server.

A trace server started in this way will simply display the formatted trace messages the Erlang shell (that is, using io:format/2). See tracer/2 for a description of how the trace message handler can be customized.

To start a similar tracer on a remote node, use n/1.

tracer(Type, Data)

-spec tracer(port, PortGenerator) -> {ok, pid()} | {error, Error :: term()}
                when PortGenerator :: fun(() -> port());
            (process, HandlerSpec) -> {ok, pid()} | {error, Error :: term()}
                when
                    HandlerSpec :: {HandlerFun, InitialData :: term()},
                    HandlerFun :: fun((Event :: term(), Data :: term()) -> NewData :: term());
            (module, ModuleSpec) -> {ok, pid()} | {error, Error :: term()}
                when
                    ModuleSpec :: fun(() -> {TracerModule, TracerState}) | {TracerModule, TracerState},
                    TracerModule :: atom(),
                    TracerState :: term();
            (file, Filename) -> {ok, pid()} | {error, Error :: term()} when Filename :: file:name_all().

Starts a tracer server with additional parameters on the local node.

Type indicates how trace messages should be handled:

process - by a receiving process

port - by a port; see trace_port/2

module - by a tracer module; see erl_tracer

file - by printing them to a file

If Type is process, Data should be a message handler function (HandlerSpec). The handler function, which should be a fun taking two arguments, will be called for each trace message, with the first argument containing the message as it is and the second argument containing the return value from the last invocation of the fun. The initial value of the second parameter is specified in the InitialData part of the HandlerSpec. The HandlerFun can choose any appropriate action to take when invoked, and can save a state for the next invocation by returning it.

If Type is port, then the second parameter should be a fun which takes no arguments and returns a newly opened trace port when called. Such a fun is preferably generated by calling trace_port/2.

If Type is module, Data should be either a tuple describing the erl_tracer module to be used for tracing and the state to be used for that tracer module, or a fun returning that kind of tuple.

if Type is file, Data should be a filename specifying a file where all the traces are to be printed.

If an error is returned, it can either be because a tracer server is already running ({error,already_started}), or because HandlerFun raised an exception.

To start a similar tracer on a remote node, use tracer/3.

tracer(Nodename, Type, Data)

-spec tracer(Nodename :: node(), Type :: term(), Data :: term()) ->
                {ok, Nodename :: node()} | {error, Reason :: term()}.

This function is equivalent to tracer/2, but acts on the given node.

A tracer is started on the node (Nodename) and the node is added to the list of traced nodes.

Note

This function is not equivalent to n/1. While n/1 starts a process tracer which redirects all trace information to a process tracer on the local node (that is, the trace control node), tracer/3 starts any type of tracer, independent of the type of tracer on the trace control node.

For details, see tracer/2.

wtp(Name)

-spec wtp(Name) -> ok | {error, IOError} when Name :: string(), IOError :: term().

Saves all match specifications saved during the session (by calls to tp/2 or tpl/2), as well as built-in match specifications, in a text file with the name designated by Name.

wtp stands for write trace patterns.

The format of the file is textual, which means that it can be edited with a text editor, and then restored with rtp/1.

Each match specification in the file ends with a period (.) and new (syntactically correct) match specifications can be added to the file manually.

The function returns ok, or an error tuple where the second element indicates the reason that writing the file failed.

dyntrace

Interface to dynamic tracing

This module implements interfaces to dynamic tracing, should such be compiled into the virtual machine. For a standard and/or commercial build, no dynamic tracing is available, in which case none of the functions in this module is usable or give any effect.

Should dynamic tracing be enabled in the current build, either by configuring with ./configure --with-dynamic-trace=dtrace or with ./configure --with-dynamic-trace=systemtap, the module can be used for two things:

Trigger the user-probe user_trace_i4s4 in the NIF library dyntrace.so by calling dyntrace:p/{1,2,3,4,5,6,7,8}.

Set a user specified tag that will be present in the trace messages of both the efile_drv and the user-probe mentioned above.

Both building with dynamic trace probes and using them is experimental and unsupported by Erlang/OTP. It is included as an option for the developer to trace and debug performance issues in their systems.

The original implementation is mostly done by Scott Lystiger Fritchie as an Open Source Contribution and it should be viewed as such even though the source for dynamic tracing as well as this module is included in the main distribution. However, the ability to use dynamic tracing of the virtual machine is a very valuable contribution which OTP has every intention to maintain as a tool for the developer.

How to write d programs or systemtap scripts can be learned from books and from a lot of pages on the Internet. This manual page does not include any documentation about using the dynamic trace tools of respective platform. However, the examples directory of the runtime_tools application contains comprehensive examples of both d and systemtap programs that will help you get started. Another source of information is the dtrace and systemtap chapters in the Runtime Tools Users' Guide.

Summary

Types

probe_arg()

Functions

available()

This function uses the NIF library to determine if dynamic tracing is available.

get_tag()

This function returns the user tag set in the current process. If no tag is set or dynamic tracing is not available, it returns undefined.

p()

Calling this function triggers the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing only the user tag and zeroes/empty strings in all other fields.

p(Arg)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer or string parameter in the first integer/string field.

p(Arg1, Arg2)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Arg1, Arg2, Arg3)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Arg1, Arg2, Arg3, Arg4)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Int, Arg1, Arg2, Arg3, String)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Int1, Int2, Arg1, Arg2, String1, String2)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Int1, Int2, Int3, Arg, String1, String2, String4)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

p(Int1, Int2, Int3, Int4, String1, String2, String3, String4)

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing all the integer() and string() parameters provided, as well as any user tag set in the current process.

put_tag(Item)

This function sets the user tag of the current process.

restore_tag(TagData)

Restores the previous state of user tags and their spreading as it was before a call to spread_tag/1.

spread_tag(boolean())

This function controls if user tags are to be spread to other processes with the next message.

Types

probe_arg()

(not exported) (since OTP R15B01)

-type probe_arg() :: integer() | iolist().

Functions

available()

(since OTP R15B01)

-spec available() -> true | false.

This function uses the NIF library to determine if dynamic tracing is available.

This function will throw an exception if the dyntrace NIF library could not be loaded by the on_load function in this module.

Use erlang:system_info(dynamic_trace) to determine whether the run-time system supports dynamic tracing.

get_tag()

(since OTP R15B01)

-spec get_tag() -> binary() | undefined.

This function returns the user tag set in the current process. If no tag is set or dynamic tracing is not available, it returns undefined.

This function returns the user tag set in the current process or, if no user tag is present, the last user tag sent to the process together with a message (in the same way as sequential trace tokens are spread to other processes together with messages. For an explanation of how user tags can be spread together with messages, see spread_tag/1. If no tag is found or dynamic tracing is not available, it returns undefined

p()

(since OTP R15B01)

-spec p() -> true | false | error | badarg.

Calling this function triggers the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing only the user tag and zeroes/empty strings in all other fields.

p(Arg)

(since OTP R15B01)

-spec p(probe_arg()) -> true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer or string parameter in the first integer/string field.

p(Arg1, Arg2)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg()) -> true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

That is, the following calls work:

dyntrace:p(1, "Hello")

dyntrace:p(1, 1)

dyntrace:p("Hello", "Again")

The following call is invalid because the string argument comes before the integer argument:

dyntrace:p("Hello", 1)

p(Arg1, Arg2, Arg3)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg(), probe_arg()) -> true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

p(Arg1, Arg2, Arg3, Arg4)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg(), probe_arg(), probe_arg()) -> true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

p(Int, Arg1, Arg2, Arg3, String)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg()) ->
           true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

There can be no more than four parameters of each type, so the first parameter must be of type integer() and the last parameter of type string().

p(Int1, Int2, Arg1, Arg2, String1, String2)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg()) ->
           true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

There can be no more than four parameters of each type, so the first two parameters must be of type integer() and the last two of type string().

p(Int1, Int2, Int3, Arg, String1, String2, String4)

(since OTP R15B01)

-spec p(probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg(), probe_arg()) ->
           true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing the user tag and the integer() or string() parameters as the first fields of their respective type.

integer() parameters should be put before any string() parameters.

There can be no more than four parameters of each type, so the first three parameters must be of type integer() and the last three of type string().

p(Int1, Int2, Int3, Int4, String1, String2, String3, String4)

(since OTP R15B01)

-spec p(probe_arg(),
        probe_arg(),
        probe_arg(),
        probe_arg(),
        probe_arg(),
        probe_arg(),
        probe_arg(),
        probe_arg()) ->
           true | false | error | badarg.

Calling this function will trigger the "user" trace probe user_trace_i4s4 in the dyntrace NIF module, sending a trace message containing all the integer() and string() parameters provided, as well as any user tag set in the current process.

put_tag(Item)

(since OTP R15B01)

-spec put_tag(undefined | iodata()) -> binary() | undefined.

This function sets the user tag of the current process.

The user tag is a binary(), but can be specified as any iodata(), which is automatically converted to a binary by this function.

The user tag is provided to the user probes triggered by calls top dyntrace:p/{1,2,3,4,5,6,7,8} as well as probes in the efile driver. In the future, user tags might be added to more probes.

The old user tag (if any) is returned, or undefined if no user tag was present, or dynamic tracing is not enabled.

restore_tag(TagData)

(since OTP R15B01)

-spec restore_tag(true | {non_neg_integer(), binary() | []}) -> true.

Restores the previous state of user tags and their spreading as it was before a call to spread_tag/1.

Note that the restoring is not limited to the same process; one can utilize this to turn off spreding in one process and restore it in a newly created process that is is actually going to send messages:

f() ->
    TagData = dyntrace:spread_tag(false),
    spawn(fun() ->
             dyntrace:restore_tag(TagData),
             do_something()
          end),
    do_something_else(),
    dyntrace:restore_tag(TagData).

Correctly handling user tags and their spreading might take some effort, as Erlang programs tend to send and receive messages so that sometimes the user tag gets lost due to various things, like double receives or communication with a port (ports do not handle user tags, in the same way as they do not handle regular sequential trace tokens).

spread_tag(boolean())

(since OTP R15B01)

-spec spread_tag(boolean()) -> true | {non_neg_integer(), binary() | []}.

This function controls if user tags are to be spread to other processes with the next message.

Spreading of user tags work like spreading of sequential trace tokens, so that a received user tag will be active in the process until the next message arrives (if that message does not also contain the user tag).

This functionality is used when a client process communicates with a file i/o-server to spread the user tag to the I/O-server and then down to the efile driver. By using spread_tag/1 and restore_tag/1, one can enable or disable spreading of user tags to other processes and then restore the previous state of the user tag. The TagData returned from this call contains all previous information so the state (including any previously spread user tags) will be completely restored by a later call to restore_tag/1.

The file module already spreads tags, so there is no need to manually call this function to get user tags spread to the efile driver through that module.

The most use of this function would be if one, for example, uses the io module to communicate with an I/O-server for a regular file, such as in the following example:

f() ->
   {ok, F} = file:open("test.tst", [write]),
   Saved = dyntrace:spread_tag(true),
   io:format(F, "Hello world!", []),
   dyntrace:restore_tag(Saved),
   file:close(F).

In this example, any user tag set in the calling process will be spread to the I/O-server when the io:format/3 call is done.

instrument

Analysis and Utility Functions for Instrumentation

The module instrument contains support for studying the resource usage in an Erlang runtime system. Currently, only the allocation of memory can be studied.

Note

Since this module inspects internal details of the runtime system it may differ greatly from one version to another. We make no compatibility guarantees in this module.

See Also

erts_alloc(3), erl(1)

Summary

Types

allocation_origin()

allocation_summary()

A summary of allocated block sizes (including their headers) grouped by their Origin and Type.

block_histogram()

A histogram of block sizes where each interval's upper bound is twice as high as the one before it.

carrier_info_list()

AllocatorType is the type of the allocator that employs this carrier.

Functions

allocations()

Equivalent to allocations(#{}).

allocations(Options)

Returns a summary of all tagged allocations in the system, optionally filtered by allocator type and scheduler id.

carriers()

Equivalent to carriers(#{}).

carriers(Options)

Returns a summary of all carriers in the system, optionally filtered by allocator type and scheduler id.

Types

allocation_origin()

(not exported)

-type allocation_origin() :: atom() | mfa() | pid() | port().

allocation_summary()

(not exported)

-type allocation_summary() ::
          {HistogramStart :: non_neg_integer(),
           UnscannedSize :: non_neg_integer(),
           Allocations :: #{Origin :: allocation_origin() => #{Type :: atom() => block_histogram()}}}.

A summary of allocated block sizes (including their headers) grouped by their Origin and Type.

Origin is generally which NIF or driver that allocated the blocks, or 'system' if it could not be determined.

Type is the allocation category that the blocks belong to, e.g. db_term, message or binary. The categories correspond to those in erl_alloc.types.

If one or more carriers could not be scanned in full without harming the responsiveness of the system, UnscannedSize is the number of bytes that had to be skipped.

block_histogram()

(not exported)

-type block_histogram() :: tuple().

A histogram of block sizes where each interval's upper bound is twice as high as the one before it.

The upper bound of the first interval is provided by the function that returned the histogram, and the last interval has no upper bound.

For example, the histogram below has 40 (message) blocks between 128-256 bytes in size, 78 blocks between 256-512 bytes,2 blocks between 512-1024 bytes, and 2 blocks between 1-2KB.

> instrument:allocations(#{ histogram_start => 128, histogram_width => 15 }).
{ok, {128, 0, #{ message => {0,40,78,2,2,0,0,0,0,0,0,0,0,0,0}, ... } }}

carrier_info_list()

(not exported)

-type carrier_info_list() ::
          {HistogramStart :: non_neg_integer(),
           Carriers ::
               [{AllocatorType :: atom(),
                 InPool :: boolean(),
                 TotalSize :: non_neg_integer(),
                 UnscannedSize :: non_neg_integer(),
                 Allocations ::
                     [{Type :: atom(), Count :: non_neg_integer(), Size :: non_neg_integer()}],
                 FreeBlocks :: block_histogram()}]}.

AllocatorType is the type of the allocator that employs this carrier.

InPool is whether the carrier is in the migration pool.

TotalSize is the total size of the carrier, including its header.

Allocations is a summary of the allocated blocks in the carrier. Note that carriers may contain multiple different block types when carrier pools are shared between different allocator types (see the erts_alloc documentation for more details).

FreeBlocks is a histogram of the free block sizes in the carrier.

If the carrier could not be scanned in full without harming the responsiveness of the system, UnscannedSize is the number of bytes that had to be skipped.

Functions

allocations()

(since OTP 21.0)

-spec allocations() -> {ok, Result} | {error, Reason}
                     when Result :: allocation_summary(), Reason :: not_enabled.

Equivalent to allocations(#{}).

allocations(Options)

(since OTP 21.0)

-spec allocations(Options) -> {ok, Result} | {error, Reason}
                     when
                         Result :: allocation_summary(),
                         Reason :: not_enabled,
                         Options ::
                             #{scheduler_ids => [non_neg_integer()],
                               allocator_types => [atom()],
                               histogram_start => pos_integer(),
                               histogram_width => pos_integer(),
                               flags => [per_process | per_port | per_mfa]}.

Returns a summary of all tagged allocations in the system, optionally filtered by allocator type and scheduler id.

Only binaries and allocations made by NIFs and drivers are tagged by default, but this can be configured an a per-allocator basis with the +M<S>atags emulator option.

If the specified allocator types are not enabled, the call will fail with {error, not_enabled}.

The following options can be used:

allocator_types - The allocator types that will be searched.

Specifying a specific allocator type may lead to strange results when carrier migration between different allocator types has been enabled: you may see unexpected types (e.g. process heaps when searching binary_alloc), or fewer blocks than expected if the carriers the blocks are on have been migrated out to an allocator of a different type.

Defaults to all alloc_util allocators.

scheduler_ids - The scheduler ids whose allocator instances will be searched. A scheduler id of 0 will refer to the global instance that is not tied to any particular scheduler. Defaults to all schedulers and the global instance.

histogram_start - The upper bound of the first interval in the allocated block size histograms. Defaults to 128.

histogram_width - The number of intervals in the allocated block size histograms. Defaults to 18.

flags - Controls how to group the output, for example showing allocations on a per-process basis (when possible) rather than only a NIF/driver-basis. Defaults to [].

Example:

> instrument:allocations(#{ histogram_start => 128, histogram_width => 15 }).
{ok,{128,0,
     #{udp_inet =>
           #{driver_event_state => {0,0,0,0,0,0,0,0,0,1,0,0,0,0,0}},
       system =>
           #{heap => {0,0,0,0,20,4,2,2,2,3,0,1,0,0,1},
             db_term => {271,3,1,52,80,1,0,0,0,0,0,0,0,0,0},
             code => {0,0,0,5,3,6,11,22,19,20,10,2,1,0,0},
             binary => {18,0,0,0,7,0,0,1,0,0,0,0,0,0,0},
             message => {0,40,78,2,2,0,0,0,0,0,0,0,0,0,0},
             ... }
       spawn_forker =>
           #{driver_select_data_state =>
                 {1,0,0,0,0,0,0,0,0,0,0,0,0,0,0}},
       ram_file_drv => #{drv_binary => {0,0,0,0,0,0,1,0,0,0,0,0,0,0,0}},
       prim_file =>
           #{process_specific_data => {2,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
             nif_trap_export_entry => {0,4,0,0,0,0,0,0,0,0,0,0,0,0,0},
             monitor_extended => {0,1,0,0,0,0,0,0,0,0,0,0,0,0,0},
             drv_binary => {0,0,0,0,0,0,1,0,3,5,0,0,0,1,0},
             binary => {0,4,0,0,0,0,0,0,0,0,0,0,0,0,0}},
       prim_buffer =>
           #{nif_internal => {0,4,0,0,0,0,0,0,0,0,0,0,0,0,0},
             binary => {0,4,0,0,0,0,0,0,0,0,0,0,0,0,0}}}}}

carriers()

(since OTP 21.0)

-spec carriers() -> {ok, Result} | {error, Reason}
                  when Result :: carrier_info_list(), Reason :: not_enabled.

Equivalent to carriers(#{}).

carriers(Options)

(since OTP 21.0)

-spec carriers(Options) -> {ok, Result} | {error, Reason}
                  when
                      Result :: carrier_info_list(),
                      Reason :: not_enabled,
                      Options ::
                          #{scheduler_ids => [non_neg_integer()],
                            allocator_types => [atom()],
                            histogram_start => pos_integer(),
                            histogram_width => pos_integer()}.

Returns a summary of all carriers in the system, optionally filtered by allocator type and scheduler id.

If the specified allocator types are not enabled, the call will fail with {error, not_enabled}.

The following options can be used:

allocator_types - The allocator types that will be searched. Defaults to all alloc_util allocators.

scheduler_ids - The scheduler ids whose allocator instances will be searched. A scheduler id of 0 will refer to the global instance that is not tied to any particular scheduler. Defaults to all schedulers and the global instance.

histogram_start - The upper bound of the first interval in the free block size histograms. Defaults to 512.

histogram_width - The number of intervals in the free block size histograms. Defaults to 14.

Example:

> instrument:carriers(#{ histogram_start => 512, histogram_width => 8 }).
{ok,{512,
     [{driver_alloc,false,262144,0,
                    [{driver_alloc,1,32784}],
                    {0,0,0,0,0,0,0,1}},
      {binary_alloc,false,32768,0,
                    [{binary_alloc,15,4304}],
                    {3,0,0,0,1,0,0,0}},
      {...}|...]}}

msacc

Convenience functions for microstate accounting

This module implements some convenience functions for analyzing microstate accounting data. For details about how to use the basic API and what the different states represent, see erlang:statistics(microstate_accounting).

Basic Scenario

1> msacc:start(1000).
ok
2> msacc:print().
Average thread real-time    : 1000513 us
Accumulated system run-time :    2213 us
Average scheduler run-time  :    1076 us

        Thread      aux check_io emulator       gc    other     port    sleep

Stats per thread:
     async( 0)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
     async( 1)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
       aux( 1)    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.99%
 scheduler( 1)    0.00%    0.03%    0.13%    0.00%    0.01%    0.00%   99.82%
 scheduler( 2)    0.00%    0.00%    0.00%    0.00%    0.03%    0.00%   99.97%

Stats per type:
         async    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%  100.00%
           aux    0.00%    0.00%    0.00%    0.00%    0.00%    0.00%   99.99%
     scheduler    0.00%    0.02%    0.06%    0.00%    0.02%    0.00%   99.89%
ok

This first command enables microstate accounting for 1000 milliseconds. See start/0, stop/0, reset/0, and start/1 for more details. The second command prints the statistics gathered during that time. First three general statistics are printed.

Average real-time - The average time spent collecting data in the threads. This should be close to the time which data was collected.

System run-time - The total run-time of all threads in the system. This is what you get if you call msacc:stats(total_runtime,Stats).

Average scheduler run-time - The average run-time for the schedulers. This is the average amount of time the schedulers did not sleep.

Then one column per state is printed with a the percentage of time this thread spent in the state out of it's own real-time. After the thread specific time, the accumulated time for each type of thread is printed in a similar format.

Since we have the average real-time and the percentage spent in each state we can easily calculate the time spent in each state by multiplying Average thread real-time with Thread state %, that is, to get the time Scheduler 1 spent in the emulator state we do 1000513us * 0.13% = 1300us.

Summary

Types

msacc_data()

msacc_data_counters()

A map containing the different microstate accounting states and the number of microseconds spent in it.

msacc_data_thread()

msacc_id()

msacc_print_options()

The different options that can be given to print/2.

msacc_state()

The different states that a thread can be in. See erlang:statistics(microstate_accounting) for details.

msacc_stats()

msacc_stats_counters()

A map containing the different microstate accounting states. Each value in the map contains another map with the percentage of time that this thread has spent in the specific state. Both the percentage of system time and the time for that specific thread is part of the map.

msacc_stats_thread()

A map containing information about a specific thread. The percentages in the map can be either run-time or real-time depending on if runtime or realtime was requested from stats/2. system is the percentage of total system time for this specific thread.

msacc_type()

Functions

available()

This function checks whether microstate accounting is available or not.

from_file(Filename)

Read a file dump produced by to_file(Filename).

print()

Prints the current microstate accounting to standard out. Equivalent to msacc:print(msacc:stats(), #{}).

print(DataOrStats)

Equivalent to print(DataOrStats, #{}).

print(DataOrStats, Options)

Print the given microstate statistics values to standard out. With many states this can be verbose. See the top of this reference manual for a brief description of what the fields mean.

print(FileOrDevice, DataOrStats, Options)

Print the given microstate statistics values to the given file or device. The other arguments behave the same way as for print/2.

reset()

Reset microstate accounting counters. Returns whether is was enabled or disabled.

start()

Start microstate accounting. Returns whether it was previously enabled or disabled.

start(Time)

Resets all counters and then starts microstate accounting for the given milliseconds.

stats()

Returns a runtime system independent version of the microstate statistics data presented by erlang:statistics(microstate_accounting). All counters have been normalized to be in microsecond resolution.

stats/2

Returns the system time for the given microstate statistics values. System time is the accumulated time of all threads.

stop()

Stop microstate accounting. Returns whether is was previously enabled or disabled.

to_file(Filename)

Dumps the current microstate statistics counters to a file that can be parsed with file:consult/1.

Types

msacc_data()

(not exported) (since OTP 19.0)

-type msacc_data() :: [msacc_data_thread()].

msacc_data_counters()

(not exported) (since OTP 19.0)

-type msacc_data_counters() :: #{msacc_state() => non_neg_integer()}.

A map containing the different microstate accounting states and the number of microseconds spent in it.

msacc_data_thread()

(not exported) (since OTP 19.0)

-type msacc_data_thread() ::
          #{'$type' := msacc_data,
            type := msacc_type(),
            id := msacc_id(),
            counters := msacc_data_counters()}.

msacc_id()

(not exported) (since OTP 19.0)

-type msacc_id() :: non_neg_integer().

msacc_print_options()

(not exported) (since OTP 19.0)

-type msacc_print_options() :: #{system => boolean()}.

The different options that can be given to print/2.

msacc_state()

(not exported) (since OTP 19.0)

-type msacc_state() ::
          alloc | aux | bif | busy_wait | check_io | emulator | ets | gc | gc_fullsweep | nif | other |
          port | send | sleep | timers.

The different states that a thread can be in. See erlang:statistics(microstate_accounting) for details.

msacc_stats()

(not exported) (since OTP 19.0)

-type msacc_stats() :: [msacc_stats_thread()].

msacc_stats_counters()

(not exported) (since OTP 19.0)

-type msacc_stats_counters() :: #{msacc_state() => #{thread := float(), system := float()}}.

A map containing the different microstate accounting states. Each value in the map contains another map with the percentage of time that this thread has spent in the specific state. Both the percentage of system time and the time for that specific thread is part of the map.

msacc_stats_thread()

(not exported) (since OTP 19.0)

-type msacc_stats_thread() ::
          #{'$type' := msacc_stats,
            type := msacc_type(),
            id := msacc_id(),
            system := float(),
            counters := msacc_stats_counters()}.

A map containing information about a specific thread. The percentages in the map can be either run-time or real-time depending on if runtime or realtime was requested from stats/2. system is the percentage of total system time for this specific thread.

msacc_type()

(not exported) (since OTP 19.0)

-type msacc_type() :: aux | async | dirty_cpu_scheduler | dirty_io_scheduler | poll | scheduler.

Functions

available()

(since OTP 19.0)

-spec available() -> boolean().

This function checks whether microstate accounting is available or not.

from_file(Filename)

(since OTP 19.0)

-spec from_file(Filename) -> msacc_data() when Filename :: file:name_all().

Read a file dump produced by to_file(Filename).

print()

(since OTP 19.0)

-spec print() -> ok.

Prints the current microstate accounting to standard out. Equivalent to msacc:print(msacc:stats(), #{}).

print(DataOrStats)

(since OTP 19.0)

-spec print(DataOrStats) -> ok when DataOrStats :: msacc_data() | msacc_stats().

Equivalent to print(DataOrStats, #{}).

print(DataOrStats, Options)

(since OTP 19.0)

-spec print(DataOrStats, Options) -> ok
               when DataOrStats :: msacc_data() | msacc_stats(), Options :: msacc_print_options().

Print the given microstate statistics values to standard out. With many states this can be verbose. See the top of this reference manual for a brief description of what the fields mean.

It is possible to print more specific types of statistics by first manipulating the DataOrStats using stats/2. For instance if you want to print the percentage of run-time for each thread you can do:

msacc:print(msacc:stats(runtime, msacc:stats())).

If you want to only print run-time per thread type you can do:

msacc:print(msacc:stats(type, msacc:stats(runtime, msacc:stats()))).

Options

system - Print percentage of time spent in each state out of system time as well as thread time. Default: false.

print(FileOrDevice, DataOrStats, Options)

(since OTP 19.0)

-spec print(FileOrDevice, DataOrStats, Options) -> ok
               when
                   FileOrDevice :: file:filename() | io:device(),
                   DataOrStats :: msacc_data() | msacc_stats(),
                   Options :: msacc_print_options().

Print the given microstate statistics values to the given file or device. The other arguments behave the same way as for print/2.

reset()

(since OTP 19.0)

-spec reset() -> boolean().

Reset microstate accounting counters. Returns whether is was enabled or disabled.

start()

(since OTP 19.0)

-spec start() -> boolean().

Start microstate accounting. Returns whether it was previously enabled or disabled.

start(Time)

(since OTP 19.0)

-spec start(Time) -> true when Time :: timeout().

Resets all counters and then starts microstate accounting for the given milliseconds.

stats()

(since OTP 19.0)

-spec stats() -> msacc_data().

Returns a runtime system independent version of the microstate statistics data presented by erlang:statistics(microstate_accounting). All counters have been normalized to be in microsecond resolution.

stats/2

(since OTP 19.0)

-spec stats(Analysis, Stats) -> non_neg_integer()
               when Analysis :: system_realtime | system_runtime, Stats :: msacc_data();
           (Analysis, Stats) -> msacc_stats() when Analysis :: realtime | runtime, Stats :: msacc_data();
           (Analysis, StatsOrData) -> msacc_data() | msacc_stats()
               when Analysis :: type, StatsOrData :: msacc_data() | msacc_stats().

Returns the system time for the given microstate statistics values. System time is the accumulated time of all threads.

realtime - Returns all time recorded for all threads.

runtime - Returns all time spent doing work for all threads, i.e. all time not spent in the sleep state.

Returns fractions of real-time or run-time spent in the various threads from the given microstate statistics values.

Returns a list of microstate statistics values where the values for all threads of the same type has been merged.

stop()

(since OTP 19.0)

-spec stop() -> boolean().

Stop microstate accounting. Returns whether is was previously enabled or disabled.

to_file(Filename)

(since OTP 19.0)

-spec to_file(Filename) -> ok | {error, file:posix()} when Filename :: file:name_all().

Dumps the current microstate statistics counters to a file that can be parsed with file:consult/1.

scheduler

Measure scheduler utilization

This module contains utility functions for easy measurement and calculation of scheduler utilization. It act as a wrapper around the more primitive API erlang:statistics(scheduler_wall_time).

The simplest usage is to call the blocking scheduler:utilization(Seconds).

For non blocking and/or continuous calculation of scheduler utilization, the recommended usage is:

First call erlang:system_flag(scheduler_wall_time, true) to enable scheduler wall time measurements.

Call get_sample/0 to collect samples with some time in between.

Call utilization/2 to calculate the scheduler utilization in the interval between two samples.

When done call erlang:system_flag(scheduler_wall_time, false) to disable scheduler wall time measurements and avoid unecessary CPU overhead.

To get correct values from utilization/2, it is important that scheduler_wall_time is kept enabled during the entire interval between the two samples. To ensure this, the process that called erlang:system_flag(scheduler_wall_time, true) must be kept alive, as scheduler_wall_time will automatically be disabled if it terminates.

Summary

Types

sched_id()

sched_sample()

sched_type()

sched_util_result()

A list of tuples containing results for individual schedulers as well as aggregated averages. Util is the scheduler utilization as a floating point value between 0.0 and 1.0. Percent is the same utilization as a more human readable string expressed in percent.

Functions

get_sample()

Returns a scheduler utilization sample for normal and dirty-cpu schedulers. Returns undefined if system flag scheduler_wall_time has not been enabled.

get_sample_all()

Return a scheduler utilization sample for all schedulers, including dirty-io schedulers. Returns undefined if system flag scheduler_wall_time has not been enabled.

sample()

Return a scheduler utilization sample for normal and dirty-cpu schedulers. Will call erlang:system_flag(scheduler_wall_time, true) first if not already already enabled.

sample_all()

Return a scheduler utilization sample for all schedulers, including dirty-io schedulers. Will call erlang:system_flag(scheduler_wall_time, true) first if not already already enabled.

utilization/1

Measure utilization for normal and dirty-cpu schedulers during Seconds seconds, and then return the result.

utilization(Sample1, Sample2)

Calculates scheduler utilizations for the time interval between the two samples obtained from calling get_sample/0 or get_sample_all/0.

Types

sched_id()

(not exported) (since OTP 21.0)

-type sched_id() :: integer().

sched_sample()

(since OTP 21.0)

-opaque sched_sample()

sched_type()

(not exported) (since OTP 21.0)

-type sched_type() :: normal | cpu | io.

sched_util_result()

(not exported) (since OTP 21.0)

-type sched_util_result() ::
          [{sched_type(), sched_id(), float(), string()} |
           {total, float(), string()} |
           {weighted, float(), string()}].

A list of tuples containing results for individual schedulers as well as aggregated averages. Util is the scheduler utilization as a floating point value between 0.0 and 1.0. Percent is the same utilization as a more human readable string expressed in percent.

{normal, SchedulerId, Util, Percent} - Scheduler utilization of a normal scheduler with number SchedulerId. Schedulers that are not online will also be included. Online schedulers have the lowest SchedulerId.

{cpu, SchedulerId, Util, Percent} - Scheduler utilization of a dirty-cpu scheduler with number SchedulerId.

{io, SchedulerId, Util, Percent} - Scheduler utilization of a dirty-io scheduler with number SchedulerId. This tuple will only exist if both samples were taken with sample_all/0.

{total, Util, Percent} - Total utilization of all normal and dirty-cpu schedulers.

{weighted, Util, Percent} - Total utilization of all normal and dirty-cpu schedulers, weighted against maximum amount of available CPU time.

Functions

get_sample()

(since OTP 24.3)

-spec get_sample() -> sched_sample() | undefined.

Returns a scheduler utilization sample for normal and dirty-cpu schedulers. Returns undefined if system flag scheduler_wall_time has not been enabled.

get_sample_all()

(since OTP 24.3)

-spec get_sample_all() -> sched_sample() | undefined.

Return a scheduler utilization sample for all schedulers, including dirty-io schedulers. Returns undefined if system flag scheduler_wall_time has not been enabled.

sample()

(since OTP 21.0)

-spec sample() -> sched_sample().

Return a scheduler utilization sample for normal and dirty-cpu schedulers. Will call erlang:system_flag(scheduler_wall_time, true) first if not already already enabled.

Note

This function is not recommended as there is no way to detect if scheduler_wall_time already was enabled or not. If scheduler_wall_time has been disabled between two samples, passing them to utilization/2 will yield invalid results.

Instead use get_sample/0 together with erlang:system_flag(scheduler_wall_time, _).

sample_all()

(since OTP 21.0)

-spec sample_all() -> sched_sample().

Return a scheduler utilization sample for all schedulers, including dirty-io schedulers. Will call erlang:system_flag(scheduler_wall_time, true) first if not already already enabled.

Note

This function is not recommended for same reason as sample/0. Instead use get_sample_all/0 together with erlang:system_flag(scheduler_wall_time,_).

utilization/1

(since OTP 21.0)

-spec utilization(Seconds) -> sched_util_result() when Seconds :: pos_integer();
                 (Sample) -> sched_util_result() when Sample :: sched_sample().

Measure utilization for normal and dirty-cpu schedulers during Seconds seconds, and then return the result.

Will automatically first enable and then disable scheduler_wall_time.

Calculate scheduler utilizations for the time interval from when Sample was taken and "now". The same as calling scheduler:utilization(Sample, scheduler:sample_all()).

Note

This function is not recommended as it's so easy to get invalid results without noticing. In particular do not do this:

scheduler:utilization(scheduler:sample()). % DO NOT DO THIS!

The above example takes two samples in rapid succession and calculates the scheduler utilization between them. The resulting values will probably be more misleading than informative.

Instead use scheduler:utilization/2 and call get_sample/0 to get samples with some time in between.

utilization(Sample1, Sample2)

(since OTP 21.0)

-spec utilization(Sample1, Sample2) -> sched_util_result()
                     when Sample1 :: sched_sample(), Sample2 :: sched_sample().

Calculates scheduler utilizations for the time interval between the two samples obtained from calling get_sample/0 or get_sample_all/0.

This function itself, does not need scheduler_wall_time to be enabled. However, for a correct result, scheduler_wall_time must have been enabled during the entire interval between the two samples.

system_information

System Information

Summary

Functions

sanity_check()

Performs a sanity check on the system.

to_file(FileName)

Writes miscellaneous system information to file. This information will typically be requested by the Erlang/OTP team at Ericsson AB when reporting an issue.

Functions

sanity_check()

(since OTP 17.0)

-spec sanity_check() -> ok | {failed, Failures}
                      when
                          Application :: atom(),
                          ApplicationVersion :: string(),
                          MissingRuntimeDependencies ::
                              {missing_runtime_dependencies, ApplicationVersion, [ApplicationVersion]},
                          InvalidApplicationVersion :: {invalid_application_version, ApplicationVersion},
                          InvalidAppFile :: {invalid_app_file, Application},
                          Failure ::
                              MissingRuntimeDependencies | InvalidApplicationVersion | InvalidAppFile,
                          Failures :: [Failure].

Performs a sanity check on the system.

If no issues were found, ok is returned. If issues were found, {failed, Failures} is returned. All failures found will be part of the Failures list. Currently defined Failure elements in the Failures list:

InvalidAppFile - An application has an invalid .app file. The second element identifies the application which has the invalid .app file.

InvalidApplicationVersion - An application has an invalid application version. The second element identifies the application version that is invalid.

MissingRuntimeDependencies - An application is missing runtime dependencies. The second element identifies the application (with version) that has missing dependencies. The third element contains the missing dependencies.

Note that this check use application versions that are loaded, or will be loaded when used. You might have application versions that satisfies all dependencies installed in the system, but if those are not loaded this check will fail. Of course, the system will also fail when used like this. This can happen when you have multiple branched versions of the same application installed in the system, but there does not exist a boot script identifying the correct application version.

Currently the sanity check is limited to verifying runtime dependencies found in the .app files of all applications. More checks will be introduced in the future. This implies that the return type will change in the future.

Note

An ok return value only means that sanity_check/0 did not find any issues, not that no issues exist.

to_file(FileName)

(since OTP 17.0)

-spec to_file(FileName) -> ok | {error, Reason}
                 when
                     FileName :: file:name_all(),
                     Reason :: file:posix() | badarg | terminated | system_limit.

Writes miscellaneous system information to file. This information will typically be requested by the Erlang/OTP team at Ericsson AB when reporting an issue.