9 Testing and tools

9.1 Tracing

We have instrumented our code in order to enable tracing. Running the application with tracing deactivated, causes a negligible performance overhead (an external call to a function which returns an atom). Activation of tracing does not require any recompilation of the code, since we rely on Erlang/OTP's built in support for dynamic trace activation. In our case tracing of calls to a given external function.

Event traces can be viewed in a generic message sequence chart tool, et, or as standard output (events are written to stdio).

See enable_trace, disable_trace and set_trace for more info.

9.2 Measurement and transformation

We have included some simple tool(s) for codec measurement (meas), performance tests (mstone1 and mstone2) and message transformation.

The tool(s) are located in the example/meas directory.

Requirement

Erlang/OTP, version R13B01 or later.
Version 3.11 or later of this application.
Version 1.6.10 or later of the asn1 application.
The flex libraries. Without it, the flex powered codecs cannot be used.

Meas results

The results from the measurement run (meas) is four excel-compatible textfiles:

decode_time.xls -> Decoding result
encode_time.xls -> Encoding result
total_time.xls -> Total (Decoding+encoding) result
message_size.xls -> Message size

Instruction

The tool contain four things:

The transformation module
The measurement (meas) module(s)
The mstone (mstone1 and mstone2) module(s)
The basic message file

The messages used by the different tools are contained in single message package file (see below for more info). The messages in this file is encoded with just one codec. During measurement initiation, the messages are read and then transformed to all codec formats used in the measurement.

The message transformation is done by the transformation module. It is used to transform a set of messages encoded with one codec into the other base codec's.

There are two different measurement tools:

meas:

Used to perform codec measurements. That is, to see what kind of performance can be expected by the different codecs provided by the megaco application.

The measurement is done by iterating over the decode/encode function for approx 2 seconds per message and counting the number of decodes/encodes.

Is best run by modifying the meas.sh.skel skeleton script provided by the tool.

To run it manually do the following:
```
        % erl -pa <path-megaco-ebin-dir> -pa <path-to-meas-module-dir>
        Erlang (BEAM) emulator version 5.6 [source]

        Eshell V5.7.1  (abort with ^G)
        1> megaco_codec_meas:start().
        ...
        2> halt().
        
```
or to make it even easier, assuming a measure shall be done on all the codecs (as above):
```
        % erl -noshell -pa <path-megaco-ebin-dir> \\
              -pa <path-to-meas-module-dir> \\
              -s megaco_codec_meas -s init stop
        
```
When run as above (this will take some time), the measurement process is done as follows:
```
For each codec:
    For each message:
        Read the message from the file
            Detect message version
            Measure decode
                Measure encode
          Write results, encode, decode and total, to file
        
```
mstone1 and mstone2:

These are two different SMP performance monitoring tool(s).

mstone1 creates a process for each codec config supported by the megaco application and let them run for a specific time (all at the same time), encoding and decoding megaco messages. The number of messages processed in total is the mstone1(1) value.

There are different ways to run the mstone1 tool, e.g. with or without the use of drivers, with only flex-empowered configs.

Is best run by modifying the mstone1.sh.skel skeleton script provided by the tool.

The mstone2 is similar to the mstone1 tool, but in this case, each created process makes only one run through the messages and then exits. A soon as a process exits, a new process (with the same config and messages) is created to takes its place. The number of messages processed in total is the mstone2(1) value.

Both these tools use the message package (time_test.msgs) provided with the tool(s), although it can run on any message package as long as it has the same structure.

This is simply an erlang compatible text-file with the following structure: {codec_name(), messages_list()}.

codec_name() = pretty | compact | ber | per | erlang      (how the messages are encoded)
messages_list() = [{message_name(), message()}]
message_name() = atom()
message() = binary()

The codec name is the name of the codec with which all messages in the message_list() has been encoded.

This file can be exported to a file structure by calling the export_messages function. This can be usefull if a measurement shall be done with an external tool. Exporting the messages creates a directory tree with the following structure:

<message package>/pretty/<message-files>
                  compact/
                  per/
                  ber/<message-files>
                  erlang/

The file includes both version 1, 2 and version 3 messages.

Notes

There are two basic ways to use the binary encodings: With package related name and termination id transformation (the 'native' encoding config) or without. This transformation converts package related names and termination id's to a more convenient internal form (equivalent with the decoded text message).

The transformation is done _after_ the actual decode has been done.

Furthermore, it is possible to make use of a linked in driver that performs some of the decode/encode, decode for ber and encode for per (the 'driver' encoding config).

Therefor in the tests, binary codecs are tested with four different encoding configs to determine exactly how the different options effect the performance: with transformation and without driver ([]), without transformation and without driver ([native]), with transformation and with driver ([driver]) and finally without transformation and with driver ([driver,native]).

Some of these messages are ripped from the call flow examples in an old version of the RFC and others are created to test a specific feature of megaco.

Be sure not no name the directory containing the measurement binaries starting with 'megaco-', e.g. megaco-meas. This will confuse the erlang application loader (erlang applications are named, e.g. megaco-1.0.2).