ttb

The Trace Tool Builder ttb is a base for building trace tools for distributed systems.

When using ttb, dbg shall not be used in parallel.

Types:

This function is a shortcut allowing to start a trace with one command. Each tuple in Patterns is converted to list which is in turn passed to ttb:tpl. The call:

ttb:start_trace([Node, OtherNode],
[{mod, foo, []}, {mod, bar, 2}],
{all, call},
[{file, File}, {handler,{fun myhandler/4, S}}])

is equivalent to

ttb:start_trace([Node, OtherNode], [{file, File}, {handler,{fun myhandler/4, S}}]),
ttb:tpl(mod, foo, []),
ttb:tpl(mod, bar, 2, []),
ttb:p(all, call)

This is equivalent to tracer(node()).

Types:

shell is equivalent to tracer(node(),[{file, {local, "ttb"}}, shell]).

dbg is equivalent to tracer(node(),[{shell, only}]).

This is equivalent to tracer(Nodes,[]).

Types:

This function starts a file trace port on all given nodes and also points the system tracer for sequential tracing to the same port.

The given Filename will be prefixed with the node name. Default Filename is "ttb".

File={wrap,Filename,Size,Count} can be used if the size of the trace logs must be limited. Default values are Size=128*1024 and Count=8.

When tracing diskless nodes, ttb must be started from an external "trace control node" with disk access, and Client must be {local, File}. All trace information is then sent to the trace control node where it is written to file.

The process_info option indicates if process information should be collected. If PI = true (which is default), each process identifier Pid is replaced by a tuple {Pid,ProcessInfo,Node}, where ProcessInfo is the process' registered name its globally registered name, or its initial function. It is possible to turn off this functionality by setting PI = false.

The {shell, ShellSpec} option indicates that the trace messages should be printed on the console as they are received by the tracing process. This implies {local, File} trace client. If the ShellSpec is only (instead of true), no trace logs are stored.

The shell option is a shortcut for {shell, true}.

The timer option indicates that the trace should be automatically stopped after MSec milliseconds. StopOpts are passed to ttb:stop/2 command if specified (default is []). Note that the timing is approximate, as delays related to network communication are always present. The timer starts after ttb:p/2 is issued, so you can set up your trace patterns before.

The overload_check option allows to enable overload checking on the nodes under trace. Module:Function(check) is performed each MSec milliseconds. If the check returns true, the tracing is disabled on a given node.
Module:Function should be able to handle at least three atoms: init, check and stop. init and stop give the user a possibility to initialize and clean up the check environment.
When a node gets overloaded, it is not possible to issue ttb:p nor any command from the ttb:tp family, as it would lead to inconsistent tracing state (different trace specifications on different node).

The flush option periodically flushes all file trace port clients (see dbg:flush_trace_port/1). When enabled, the buffers are freed each MSec milliseconds. This option is not allowed with {file, {local, File}} tracing.

{resume, FetchTimeout} enables the autoresume feature. Whenever enabled, remote nodes try to reconnect to the controlling node in case they were restarted. The feature requires runtime_tools application to be started (so it has to be present in the .boot scripts if the traced nodes run with embedded erlang). If this is not possible, resume may be performed manually by starting runtime_tools remotely using rpc:call/4.
ttb tries to fetch all logs from a reconnecting node before reinitializing the trace. This has to finish within FetchTimeout milliseconds or is aborted
By default, autostart information is stored in a file called ttb_autostart.bin on each node. If this is not desired (i.e. on diskless nodes), a custom module to handle autostart information storage and retrieval can be provided by specifying ttb_autostart_module environment variable for the runtime_tools application. The module has to respond to the following API:

write_config(Data) -> ok

Store the provided data for further retrieval. It is important to realize that the data storage used must not be affected by the node crash.

read_config() -> {ok, Data} | {error, Error}

Retrieve configuration stored with write_config(Data).

delete_config() -> ok

Delete configuration stored with write_config(Data). Note that after this call any subsequent calls to read_config must return {error, Error}.

The resume option implies the default FetchTimeout, which is 10 seconds

Types:

This function sets the given trace flags on the given processes. The timestamp flag is always turned on.

Please turn to the Reference manual for module dbg for details about the possible trace flags. The parameter MatchDesc is the same as returned from dbg:p/2

Processes can be given as registered names, globally registered names or process identifiers. If a registered name is given, the flags are set on processes with this name on all active nodes.

Issuing this command starts the timer for this trace if timer option was specified with tracer/2.

These functions should be used in combination with the call trace flag for setting and clearing trace patterns. When the call trace flag is set on a process, function calls will be traced on that process if a trace pattern has been set for the called function. Trace patterns specifies how to trace a function by using match specifications. Match specifications are described in the User's Guide for the erlang runtime system erts.

These functions are equivalent to the corresponding functions in dbg, but all calls are stored in the history. The history buffer makes it easy to create config files so that the same trace environment can be setup several times, e.g. if you want to compare two test runs. It also reduces the amount of typing when using ttb from the erlang shell.

tp

Set trace pattern on global function calls

tpl

Set trace pattern on local and global function calls

ctp

Clear trace pattern on local and global function calls

ctpl

Clear trace pattern on local function calls

ctpg

Clear trace pattern on global function calls

With tp and tpl one of match specification shortcuts may be used (example: ttb:tp(foo_module, caller)). The shortcuts are:

return - for [{'_',[],[{return_trace}]}] (report the return value)

caller - for [{'_',[],[{message,{caller}}]}] (report the calling function)

{codestr, Str} - for dbg:fun2ms/1 arguments passed as strings (example: "fun(_) -> return_trace() end")

Types:

All calls to ttb is stored in the history. This function returns the current content of the history. Any entry can be re-executed with run_history/1 or stored in a config file with write_config/2/3.

Types:

Executes the given entry or entries from the history list. History can be listed with list_history/0.

Equivalent to write_config(ConfigFile,Config,[]).

Types:

This function creates or extends a config file which can be used for restoring a specific configuration later.

The content of the config file can either be fetched from the history or given directly as a list of {Mod,Func,Args}.

If the complete history is to be stored in the config file Config should be all. If only a selected number of entries from the history should be stored, Config should be a list of integers pointing out the entries to be stored.

If Opts is not given or if it is [], ConfigFile is deleted and a new file is created. If Opts = [append], ConfigFile will not be deleted. The new information will be appended at the end of the file.

Types:

Executes all entries in the given config file. Note that the history of the last trace is always available in the file named ttb_last_config.

Types:

Executes selected entries from the given config file. NumList is a list of integers pointing out the entries to be executed.

The content of a config file can be listed with list_config/1.

Note that the history of the last trace is always available in the file named ttb_last_config.

Types:

Lists all entries in the given config file.

Types:

The .ti file contains {Key,ValueList} tuples. This function adds Data to the ValueList associated with Key. All information written with this function will be included in the call to the format handler.

Equivalent to seq_trigger_ms(all)

Types:

A match specification can turn on or off sequential tracing. This function returns a match specification which turns on sequential tracing with the given Flags.

This match specification can be given as the last argument to tp or tpl. The activated Item will then become a trigger for sequential tracing. This means that if the item is called on a process with the call trace flag set, the process will be "contaminated" with the seq_trace token.

If Flags = all, all possible flags are set.

Please turn to the reference manual for the seq_trace module in the kernel application to see the possible values for SeqTraceFlag. For a description of the match_spec() syntax, please turn to the User's guide for the runtime system (erts). The chapter Match Specification in Erlang explains the general match specification "language".

Example of how to use the seq_trigger_ms/0/1 function:

(tiger@durin)5> ttb:tracer().
{ok,[tiger@durin]}
(tiger@durin)6> ttb:p(all,call).
{ok,{[all],[call]}}
(tiger@durin)7> ttb:tp(mod,func,ttb:seq_trigger_ms()).
{ok,[{matched,1},{saved,1}]}
(tiger@durin)8>         

Whenever mod:func(...) is called after this, the seq_trace token will be set on the executing process.

Equivalent to stop([]).

Types:

Stops tracing on all nodes. Logs and trace information files are sent to the trace control node and stored in a directory named ttb_upload_FileName-Timestamp, where Filename is the one provided with {file, File} during trace setup and Timestamp is of the form yyyymmdd-hhmmss. Even logs from nodes on the same machine as the trace control node are moved to this directory. The history list is saved to a file named ttb_last_config for further reference (as it will be not longer accessible through history and configuration management functions (like ttb:list_history/0).

The nofetch option indicates that trace logs shall not be collected after tracing is stopped.

The {fetch, Dir} option allows to specify the directory to fetch the data to. If the directory already exists, an error is thrown.

The format option indicates that the trace logs shall be formatted after tracing is stopped. All logs in the fetch directory will be merged. You may use {format, FormatOpts} to pass additional arguments to format/2.

The return_fetch_dir option indicates that the return value should be {stopped, Dir} and not just stopped. This implies fetch.

The et handler returned by the function may be used with format/2 or tracer/2. Example: ttb:format(Dir, [{handler, ttb:get_et_handler()}]).

Same as format(File,[]).

Types:

Reads the given binary trace log(s). The logs are processed in the order of their timestamp as long as disable_sort option is not given.

If FormatHandler = {Function,InitialState}, Function will be called for each trace message. If FormatHandler = get_et_handler(), et_viewer in the Event Tracer application (et) is used for presenting the trace log graphically. ttb provides a few different filters which can be selected from the Filter menu in the et_viewer. If FormatHandler is not given, a default handler is used which presents each trace message as a line of text.

The state returned from each call of Function is passed to the next call, even if next call is to format a message from another log file.

If Out is given, FormatHandler gets the file descriptor to Out as the first parameter.

Out is ignored if et format handler is used.

Wrap logs can be formatted one by one or all in one go. To format one of the wrap logs in a set, give the exact name of the file. To format the whole set of wrap logs, give the name with '*' instead of the wrap count. See examples in the ttb User's Guide.