Specialized Decodes

2 Specialized Decodes

When performance is of highest priority and one is interested in a limited part of the ASN.1 encoded message, before one decide what to do with the rest of it, one may want to decode only this small part. The situation may be a server that has to decide to which addressee it will send a message. The addressee may be interested in the entire message, but the server may be a bottleneck that one want to spare any unnecessary load. Instead of making two complete decodes (the normal case of decode), one in the server and one in the addressee, it is only necessary to make one specialized decode(in the server) and another complete decode(in the addressee). The following specialized decodes exclusive decode and selected decode support to solve this and similar problems.

So far this functionality is only provided when using the optimized BER_BIN version, that is when compiling with the options ber_bin and optimize. It does also work using the driver option. We have no intent to make this available on the default BER version, but maybe in the PER_BIN version (per_bin).

2.1 Exclusive Decode

The basic idea with exclusive decode is that you specify which parts of the message you want to exclude from being decoded. These parts remain encoded and are returned in the value structure as binaries. They may be decoded in turn by passing them to a certain decode_part/2 function. The performance gain is high when the message is large and you can do an exclusive decode and later on one or several decodes of the parts or a second complete decode instead of two or more complete decodes.

2.1.1 How To Make It Work

In order to make exclusive decode work you have to do the following:

First,decide the name of the function for the exclusive decode.
Second, write instructions that must consist of the name of the exclusive decode function, the name of the ASN.1 specification and a notation that tells which parts of the message structure will be excluded from decode. These instructions shall be included in a configuration file.
Third, compile with the additional option asn1config. The compiler searches for a configuration file with the same name as the ASN.1 spec but with the extension .asn1config. This configuration file is not the same as used for compilation of a set of files. See section Writing an Exclusive Decode Instruction.

2.1.2 User Interface

The run-time user interface for exclusive decode consists of two different functions. First, the function for an exclusive decode, whose name the user decides in the configuration file. Second, the compiler generates a decode_part/2 function when exclusive decode is chosen. This function decodes the parts that were left undecoded during the exclusive decode. Both functions are described below.

If the exclusive decode function has for example got the name decode_exclusive and an ASN.1 encoded message Bin shall be exclusive decoded, the call is:

{ok,Excl_Message} = 'MyModule':decode_exclusive(Bin)

The result Excl_Message has the same structure as an complete decode would have, except for the parts of the top-type that were not decoded. The undecoded parts will be on their place in the structure on the format {Type_Key,Undecoded_Value}.

Each undecoded part that shall be decoded must be fed into the decode_part/2 function,like:

{ok,Part_Message} = 'MyModule':decode_part(Type_Key,Undecoded_Value)

2.1.3 Writing an Exclusive Decode Instruction

This instruction is written in the configuration file on the format:


Exclusive_Decode_Instruction = {exclusive_decode,{Module_Name,Decode_Instructions}}.

Module_Name = atom()

Decode_Instructions = [Decode_Instruction]+

Decode_Instruction = {Exclusive_Decode_Function_Name,Type_List}

Exclusive_Decode_Function_Name = atom()

Type_List = [Top_Type,Element_List]

Element_List = [Element]+

Element = {Name,parts} |
          {Name,undecoded} |
          {Name,Element_List}

Top_Type = atom()

Name = atom()

Observe that the instruction must be a valid Erlang term ended by a dot.

In the Type_List the "path" from the top type to each undecoded sub-components is described. The top type of the path is an atom, the name of it. The action on each component/type that follows will be described by one of {Name,parts}, {Name,undecoded}, {Name,Element_List}

The use and effect of the actions are:

{Name,undecoded} Tells that the element will be left undecoded during the exclusive decode. The type of Name may be any ASN.1 type. The value of element Name will be returned as a tuple,as mentioned above , in the value structure of the top type.
{Name,parts} The type of Name may be one of SEQUENCE OF or SET OF. The action implies that the different components of Name will be left undecoded. The value of Name will be returned as a tuple, as above , where the second element is a list of binaries. That is because the representation of a SEQUENCE OF/ SET OF in Erlang is a list of its internal type. Any of the elements of this list or the entire list can be decoded by the decode_part function.
{Name,Element_List}This action is used when one or more of the sub-types of Name will be exclusive decoded.

Name in the actions above may be a component name of a SEQUENCE or a SET or a name of an alternative in a CHOICE.

2.1.4 Example

In the examples below we use the definitions from the following ASN.1 spec:

GUI DEFINITIONS AUTOMATIC TAGS ::=

BEGIN

Action ::= SEQUENCE 
 { 
   number  INTEGER DEFAULT 15,
   handle  [0] Handle DEFAULT {number  12, on  TRUE}
 }

Key ::= [11] EXPLICIT Button
Handle ::= [12] Key
Button ::= SEQUENCE 
 {
   number  INTEGER,
   on  BOOLEAN
 }

Window ::= CHOICE 
 {
   vsn INTEGER,
   status E
 }

Status ::= SEQUENCE 
 {
   state INTEGER,
   buttonList SEQUENCE OF Button,
   enabled BOOLEAN OPTIONAL,
   actions CHOICE {
     possibleActions SEQUENCE OF Action,
     noOfActions INTEGER
   }
 }


END

If Button is a top type and we want to exclude component number from decode the Type_List in the instruction in the configuration file will be ['Button',[{number,undecoded}]]. If we call the decode function decode_Button_exclusive the Decode_Instruction will be {decode_Button_exclusive,['Button',[{number,undecoded}]]}.

We also have another top type Window whose sub component actions in type Status and the parts of component buttonList shall be left undecoded. For this type we name the function decode__Window_exclusive. The whole Exclusive_Decode_Instruction configuration is as follows:

{exclusive_decode,{'GUI',
        [{decode_Window_exclusive,['Window',[{status,[{buttonList,parts},{actions,undecoded}]}]]},
         {decode_Button_exclusive,['Button',[{number,undecoded}]]}]}}.

Figure symbolizes the bytes of a Window:status message. The components buttonList and actions are excluded from decode. Only state and enabled are decoded when decode__Window_exclusive is called.

Compiling GUI.asn including the configuration file is done like:

unix> erlc -bber_bin +optimize +asn1config GUI.asn

erlang> asn1ct:compile('GUI',[ber_bin,optimize,asn1config]).

The module can be used like:


1> Button_Msg = {'Button',123,true}.
{'Button',123,true}
2> {ok,Button_Bytes} = 'GUI':encode('Button',Button_Msg).
{ok,[<<48>>,
     [6],
     [<<128>>,
      [1],
      123],
     [<<129>>,
      [1],
      255]]}
3> {ok,Exclusive_Msg_Button} = 'GUI':decode_Button_exclusive(list_to_binary(Button_Bytes)).
{ok,{'Button',{'Button_number',<<128,1,123>>},
         true}}
4> 'GUI':decode_part('Button_number',<<128,1,123>>).
{ok,123}
5> Window_Msg = {'Window',{status,{'Status',35,[{'Button',3,true},{'Button',4,false},{'Button',5,true},{'Button',6,true},{'Button',7,false},{'Button',8,true},{'Button',9,true},{'Button',10,false},{'Button',11,true},{'Button',12,true},{'Button',13,false},{'Button',14,true}],false,{possibleActions,[{'Action',16,{'Button',17,true}}]}}}}.
{'Window',{status,{'Status',35,
              [{'Button',3,true},
               {'Button',4,false},
               {'Button',5,true},
               {'Button',6,true},
               {'Button',7,false},
               {'Button',8,true},
               {'Button',9,true},
               {'Button',10,false},
               {'Button',11,true},
               {'Button',12,true},
               {'Button',13,false},
               {'Button',14,true}],
              false,
              {possibleActions,[{'Action',16,{'Button',17,true}}]}}}}
6> {ok,Window_Bytes}='GUI':encode('Window',Window_Msg).
{ok,[<<161>>,
     [127],
     [<<128>>, ...


8> {ok,{status,{'Status',Int,{Type_Key_SeqOf,Val_SEQOF},BoolOpt,{Type_Key_Choice,Val_Choice}}}}='GUI':decode_Window_status_exclusive(list_to_binary(Window_Bytes)).
{ok,{status,{'Status',35,
             {'Status_buttonList',[<<48,6,128,1,3,129,1,255>>,
                                   <<48,6,128,1,4,129,1,0>>,
                                   <<48,6,128,1,5,129,1,255>>,
                                   <<48,6,128,1,6,129,1,255>>,
                                   <<48,6,128,1,7,129,1,0>>,
                                   <<48,6,128,1,8,129,1,255>>,
                                   <<48,6,128,1,9,129,1,255>>,
                                   <<48,6,128,1,10,129,1,0>>,
                                   <<48,6,128,1,11,129,1,255>>,
                                   <<48,6,128,1,12,129,1,255>>,
                                   <<48,6,128,1,13,129,1,0>>,
                                   <<48,6,128,1,14,129,1,255>>]},
             false,
             {'Status_actions',<<163,21,160,19,48,17,2,1,16,160,12,172,10,171,8,48,6,128,1,...>>}}}}
10> 'GUI':decode_part(Type_Key_SeqOf,Val_SEQOF).
{ok,[{'Button',3,true},
     {'Button',4,false},
     {'Button',5,true},
     {'Button',6,true},
     {'Button',7,false},
     {'Button',8,true},
     {'Button',9,true},
     {'Button',10,false},
     {'Button',11,true},
     {'Button',12,true},
     {'Button',13,false},
     {'Button',14,true}]}
11> 'GUI':decode_part(Type_Key_SeqOf,hd(Val_SEQOF)).
{ok,{'Button',3,true}}
12> 'GUI':decode_part(Type_Key_Choice,Val_Choice).  
{ok,{possibleActions,[{'Action',16,{'Button',17,true}}]}}

2.2 Selective Decode

This specialized decode decodes one single subtype of a constructed value. It is the fastest method to extract one sub value. The typical use of this decode is when one want to inspect, for instance a version number,to be able to decide what to do with the entire value. The result is returned as {ok,Value} or {error,Reason}.

2.2.1 How To Make It Work

The following steps are necessary:

Write instructions in the configuration file. Including the name of a user function, the name of the ASN.1 specification and a notation that tells which part of the type will be decoded.
Compile with the additional option asn1config. The compiler searches for a configuration file with the same name as the ASN.1 spec but with the extension .asn1config. In the same file you can provide configuration specs for exclusive decode as well. The generated Erlang module has the usual functionality for encode/decode preserved and the specialized decode functionality added.

2.2.2 User Interface

The only new user interface function is the one provided by the user in the configuration file. You can invoke that function by the ModuleName:FunctionName notation.

So, if you have the following spec {selective_decode,{'ModuleName',[{selected_decode_Window,TypeList}]}} in the con-fig file, you do the selective decode by {ok,Result}='ModuleName':selected_decode_Window(EncodedBinary).

2.2.3 Writing a Selective Decode Instruction

It is possible to describe one or many selective decode functions in a configuration file, you have to use the following notation:

Selective_Decode_Instruction = {selective_decode,{Module_Name,Decode_Instructions}}.

Module_Name = atom()

Decode_Instructions = [Decode_Instruction]+

Decode_Instruction = {Selective_Decode_Function_Name,Type_List}

Selective_Decode_Function_Name = atom()

Type_List = [Top_Type|Element_List]

Element_List = Name|List_Selector

Name = atom()

List_Selector = [integer()]

Observe that the instruction must be a valid Erlang term ended by a dot.

The Module_Name is the same as the name of the ASN.1 spec, but without the extension. A Decode_Instruction is a tuple with your chosen function name and the components from the top type that leads to the single type you want to decode. Notice that you have to choose a name of your function that will not be the same as any of the generated functions. The first element of the Type_List is the top type of the encoded message. In the Element_List it is followed by each of the component names that leads to selected type. Each of the names in the Element_List must be constructed types except the last name, which can be any type.

The List_Selector makes it possible to choose one of the encoded components in a SEQUENCE OF/ SET OF. It is also possible to go further in that component and pick a sub type of that to decode. So in the Type_List: ['Window',status,buttonList,[1],number] the component buttonList has to be a SEQUENCE OF or SET OF type. In this example component number of the first of the encoded elements in the SEQUENCE OF buttonList is selected. This apply on the ASN.1 spec above.

2.2.4 Another Example

In this example we use the same ASN.1 spec as above. A valid selective decode instruction is:

{selective_decode,{'GUI',[{selected_decode_Window1,['Window',status,buttonList,[1],number]},
                          {selected_decode_Action,['Action',handle,number]},
                          {selected_decode_Window2,['Window',status,actions,possibleActions,[1],handle,number]}]}}.

The first Decode_Instruction, {selected_decode_Window1,['Window',status,buttonList,[1],number]} is commented in the previous section. The instruction {selected_decode_Action,['Action',handle,number]} picks the component number in the handle component of the type Action. If we have the value ValAction = {'Action',17,{'Button',4711,false}} the internal value 4711 should be picked by selected_decode_Action. In an Erlang terminal it looks like:

ValAction = {'Action',17,{'Button',4711,false}}.
{'Action',17,{'Button',4711,false}}
7> {ok,Bytes}='GUI':encode('Action',ValAction).
...
8> BinBytes = list_to_binary(Bytes).
<<48,18,2,1,17,160,13,172,11,171,9,48,7,128,2,18,103,129,1,0>>
9> 'GUI':selected_decode_Action(BinBytes).
{ok,4711}
10>

The third instruction, ['Window',status,actions,possibleActions,[1],handle,number], which is a little more complicated,

starts with type Window.
Picks component status of Window that is of type Status.
Then takes component actions of type Status.
Then possibleActions of the internal defined CHOICE type.
Thereafter it goes into the first component of the SEQUENCE OF by [1]. That component is of type Action.
The instruction next picks component handle.
And finally component number of the type Button.

The following figures shows which components are in the TypeList ['Window',status,actions,possibleActions,[1],handle,number]. And which part of a message that will be decoded by selected_decode_Window2.

The elements specified in the config file for selective decode of a sub-value in a Window message

Figure symbolizes the bytes of a Window:status message. Only the marked element is decoded when selected_decode_Window2 is called.

With the following example you can examine that both selected_decode_Window2 and selected_decode_Window1 decodes the intended sub-value of the value Val

1> Val = {'Window',{status,{'Status',12,
                    [{'Button',13,true},
                     {'Button',14,false},
                     {'Button',15,true},
                     {'Button',16,false}],
                    true,
                    {possibleActions,[{'Action',17,{'Button',18,false}},
                                      {'Action',19,{'Button',20,true}},
                                      {'Action',21,{'Button',22,false}}]}}}}
2> {ok,Bytes}='GUI':encode('Window',Val).
...
3> Bin = list_to_binary(Bytes).
<<161,101,128,1,12,161,32,48,6,128,1,13,129,1,255,48,6,128,1,14,129,1,0,48,6,128,1,15,129,...>>
4> 'GUI':selected_decode_Window1(Bin).
{ok,13}
5> 'GUI':selected_decode_Window2(Bin).
{ok,18}

Observe that the value feed into the selective decode functions must be a binary.

2.3 Performance

To give an indication on the possible performance gain using the specialized decodes, some measures have been performed. The relative figures in the outcome between selective, exclusive and complete decode (the normal case) depends on the structure of the type, the size of the message and on what level the selective and exclusive decodes are specified.

2.3.1 ASN.1 Specifications, Messages and Configuration

The specs GUI and MEDIA-GATEWAY-CONTROL was used in the test.

For the GUI spec the configuration looked like:

{selective_decode,{'GUI',[{selected_decode_Window1,['Window',status,buttonList,[1],number]},
                          {selected_decode_Window2,['Window',status,actions,possibleActions,[1],handle,number]}]}}.
{exclusive_decode,{'GUI',[{decode_Window_status_exclusive,['Window',[{status,[{buttonList,parts},{actions,undecoded}]}]]}]}}.

The MEDIA-GATEWAY-CONTROL configuration was:

{exclusive_decode,
  {'MEDIA-GATEWAY-CONTROL',
    [{decode_MegacoMessage_exclusive,['MegacoMessage',[{authHeader,undecoded},{mess,[{mId,undecoded},{messageBody,undecoded}]}]]}]}}.
{selective_decode,
  {'MEDIA-GATEWAY-CONTROL',
    [{decode_MegacoMessage_selective,['MegacoMessage',mess,version]}]}}.

The corresponding values were:

{'Window',{status,{'Status',12,
              [{'Button',13,true},
               {'Button',14,false},
               {'Button',15,true},
               {'Button',16,false},
               {'Button',13,true},
               {'Button',14,false},
               {'Button',15,true},
               {'Button',16,false},
               {'Button',13,true},
               {'Button',14,false},
               {'Button',15,true},
               {'Button',16,false}],
              true,
              {possibleActions,
                 [{'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}},
                  {'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}},
                  {'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}},
                  {'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}},
                  {'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}},
                  {'Action',17,{'Button',18,false}},
                  {'Action',19,{'Button',20,true}},
                  {'Action',21,{'Button',22,false}}]}}}}


{'MegacoMessage',asn1_NOVALUE,
  {'Message',1,
    {ip4Address,
      {'IP4Address',[125,125,125,111],55555}},
  {transactions,
    [{transactionReply,
      {'TransactionReply',50007,asn1_NOVALUE,
       {actionReplies,
        [{'ActionReply',0,asn1_NOVALUE,asn1_NOVALUE,
          [{auditValueReply,{auditResult,{'AuditResult',
            {'TerminationID',[],[255,255,255]},
             [{mediaDescriptor,
               {'MediaDescriptor',asn1_NOVALUE,
                {multiStream,
                 [{'StreamDescriptor',1,
                   {'StreamParms',
                    {'LocalControlDescriptor',sendRecv,asn1_NOVALUE,asn1_NOVALUE,
                     [{'PropertyParm',[0,11,0,7],[[52,48]],asn1_NOVALUE}]},
                    {'LocalRemoteDescriptor',
                     [[{'PropertyParm',[0,0,176,1],[[48]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,8],[[73,78,32,73,80,52,32,49,50,53,46,49,50,53,46,49,50,53,46,49,49,49]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,15],[[97,117,100,105,111,32,49,49,49,49,32,82,84,80,47,65,86,80,32,32,52]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,12],
                        [[112,116,105,109,101,58,51,48]],asn1_NOVALUE}]]},
                    {'LocalRemoteDescriptor',
                     [[{'PropertyParm',[0,0,176,1],[[48]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,8],[[73,78,32,73,80,52,32,49,50,52,46,49,50,52,46,49,50,52,46,50,50,50]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,15],[[97,117,100,105,111,32,50,50,50,50,32,82,84,80,47,65,86,80,32,32,52]],asn1_NOVALUE},
                       {'PropertyParm',[0,0,176,12],[[112,116,105,109,101,58,51,48]],asn1_NOVALUE}]]}}}]}}},
              {packagesDescriptor,
               [{'PackagesItem',[0,11],1},
                {'PackagesItem',[0,11],1}]},
              {statisticsDescriptor,
               [{'StatisticsParameter',[0,12,0,4],[[49,50,48,48]]},
                {'StatisticsParameter',[0,11,0,2],[[54,50,51,48,48]]},
                {'StatisticsParameter',[0,12,0,5],[[55,48,48]]},
                {'StatisticsParameter',[0,11,0,3],[[52,53,49,48,48]]},
                {'StatisticsParameter',[0,12,0,6],[[48,46,50]]},
                {'StatisticsParameter',[0,12,0,7],[[50,48]]},
                {'StatisticsParameter',[0,12,0,8],[[52,48]]}]}]}}}]}]}}}]}}}

The size of the encoded values was 458 bytes for GUI and 464 bytes for MEDIA-GATEWAY-CONTROL.

2.3.2 Results

The ASN.1 specs in the test are compiled with the options ber_bin, optimize, driver and asn1config. If the driver option had been omitted there should have been higher values for decode and decode_part.

The test program runs 10000 decodes on the value, resulting in a printout with the elapsed time in microseconds for the total number of decodes.

*Results of complete, exclusive and selective decode*
Function	Time (microseconds)	Kind of Decode	ASN.1 spec	% of time vs. complete decode
`decode_MegacoMessage_selective/1`	`374045`	`selective`	`MEDIA-GATEWAY-CONTROL`	8.3
`decode_MegacoMessage_exclusive/1`	`621107`	`exclusive`	`MEDIA-GATEWAY-CONTROL`	13.8
`decode/2`	`4507457`	`complete`	`MEDIA-GATEWAY-CONTROL`	100
`selected_decode_Window1/1`	`449585`	`selective`	`GUI`	7.6
`selected_decode_Window2/1`	`890666`	`selective`	`GUI`	15.1
`decode_Window_status_exclusive/1`	`1251878`	`exclusive`	`GUI`	21.3
`decode/2`	`5889197`	`complete`	`GUI`	100

Another interesting question is what the relation is between a complete decode, an exclusive decode followed by decode_part of the excluded parts and a selective decode followed by a complete decode. Some situations may be compared to this simulation, e.g. inspect a sub-value and later on look at the entire value. The following table shows figures from this test. The number of loops and time unit is the same as in the previous test.

*Results of complete, exclusive + decode_part and selective + complete decodes*
Actions	Function	Time (microseconds)	ASN.1 spec	% of time vs. complete decode
`complete`	`decode/2`	`4507457`	`MEDIA-GATEWAY-CONTROL`	100
`selective and complete`	`decode_MegacoMessage_selective/1`	`4881502`	`MEDIA-GATEWAY-CONTROL`	108.3
`exclusive and decode_part`	`decode_MegacoMessage_exclusive/1`	`5481034`	`MEDIA-GATEWAY-CONTROL`	112.3
`complete`	`decode/2`	`5889197`	`GUI`	100
`selective and complete`	`selected_decode_Window1/1`	`6337636`	`GUI`	107.6
`selective and complete`	`selected_decode_Window2/1`	`6795319`	`GUI`	115.4
`exclusive and decode_part`	`decode_Window_status_exclusive/1`	`6249200`	`GUI`	106.1

Other ASN.1 types and values can differ much from these figures. Therefore it is important that you, in every case where you intend to use either of these decodes, perform some tests that shows if you will benefit your purpose.

2.3.3 Comments

Generally speaking the gain of selective and exclusive decode in advance of complete decode is greater the bigger value and the less deep in the structure you have to decode. One should also prefer selective decode instead of exclusive decode if you are interested in just one single sub-value.

Another observation is that the exclusive decode followed by decode_part decodes is very attractive if the parts will be sent to different servers for decoding or if one in some cases not is interested in all parts.

The fastest selective decode are when the decoded type is a primitive type and not so deep in the structure of the top type. The selected_decode_Window2 decodes a big constructed value, which explains why this operation is relatively slow.

It may vary from case to case which combination of selective/complete decode or exclusive/part decode is the fastest.