[erlang-questions] Diameter client issue

Anders Svensson anders.otp@REDACTED
Fri Mar 29 07:19:15 CET 2013


On Thu, Mar 28, 2013 at 8:39 PM, S X <erlangprogram@REDACTED> wrote:
> Hi, Anders,
>
> Just to assure your answer with wireshark, the external server closes the
> connection after receiving the initial CER with wrong OH and OR from the
> client. But the server should response with CEA saying wrong information as
> a common sense.

Yes, it should answer with an appropriate Result-Code. If it's
rejecting the CER based on the received Origin-Host or Origin-Realm
then 3010, DIAMETER_UNKNOWN_PEER, is the best fit.

> Also, is the node format "nodename@REDACTED" is common to the diameter
> protocol and not just for the distributed erlang nodes?

nodename@REDACTED has nothing to do with the Diameter protocol: it's
the name of an Erlang node. Both Erlang and Diameter use the term
"node". You give an Erlang node a name by passing -sname or -name to
erl(1), you give a Diameter node a name by assigning it an Origin-Host
that's communicated during capabilities exchange. You choose these
names independently.

Distributed Erlang is also unrelated to the Diameter protocol. You can
read more about it here:

  http://www.erlang.org/doc/reference_manual/distributed.html

The diameter application can make use of distribution (as can your own
code) if you spread the implementation of your Diameter node across
multiple Erlang nodes, but you can also implement your Diameter node
on a single Erlang node. In both cases, your Diameter node
communicates with other Diameter nodes using the Diameter protocol
over TCP or SCTP. Distributed Erlang isn't involved.

/Anders, Erlang/OTP

> Thanks,
>
> Samuel
>
>
> On Wed, Mar 27, 2013 at 4:18 PM, Anders Svensson <anders.otp@REDACTED>
> wrote:
>>
>> Hi Samuel.
>>
>> On Wed, Mar 27, 2013 at 6:50 PM, S X <erlangprogram@REDACTED> wrote:
>> > Hi Anders & Rudolph,
>> >
>> > Thanks for your responses.
>> >
>> > "5000 sec reconnect_timer is on the long side." I guess I treated milli
>> > as
>> > micro. :)
>> >
>> > Don't worry about the logging, as long as I can see something from the
>> > erlang debugger window (By the way, I feel that the debugger for Window
>> > version is better than the one for Ubuntu). And I understand that it is
>> > kind
>> > of difficult to log everything from the diameter library.
>> >
>> > As Anders suggested, I checked on the peer side (the diameter server in
>> > this
>> > case) and it seems timed out waiting for CER and disconnecting the
>> > client.
>> >
>> > Also, I found that my wireshark wasn't set up properly to work with VPN
>> > ( I
>> > am running it with a VPN environment) and that is properly why I didn't
>> > see
>> > any packets going outside.
>> >
>> > At last, I made the communication work after changing Origin-Host and
>> > Origin-Realm on the diameter client side. And they can talk diameter
>> > now.
>> > But don't quite understand why those two options are important for the
>> > initial socket set-up? For the initial step, doesn't it just need server
>> > IP
>> > and port information? Or maybe because the special implementation on
>> > this
>> > diameter server? Is this common to a diameter application?
>>
>> Origin-Host/Realm are only used to construct an outgoing capabilities
>> CER/CEA after the transport connection is established. Are you sure
>> the server wasn't receiving your client's CER and closing the
>> connection for some reason? Changing the values of Origin-Host/Realm
>> shouldn't make a difference unless the server cares. (Which is a bit
>> unusual. If anything it's IP addresses that servers seem to like to
>> reject connections from.)
>>
>> > The other question is about diameter node, in diameter_sync module:
>> > call(Name, Req, Max, Timeout) ->
>> >     call(node(), Name, Req, Max, Timeout).
>> >
>> > By default, it is "nonode@REDACTED", is the node used just for
>> > representing
>> > the erlang run time or for other purpose? How does the node information
>> > is
>> > being used in diameter?
>>
>> nonode@REDACTED is the name of an Erlang node that isn't distributed.
>> That is, that doesn't register with epmd and doesn't talk to other
>> Erlang nodes using the Distribution Protocol. If you start a
>> distributed node (eg. erl -sname foo) then diameter can share
>> information about transport connections with diameter services running
>> on other nodes. This isn't documented (or working :) in R16B but it is
>> on the maint branch on github, on the way to R16B01. (Take a look at
>> the doc for the service_opt() share_peers in diameter(3).)
>>
>> /Anders, Erlang/OTP
>>
>>
>> > Many thanks!
>> >
>> >
>> > Samuel
>> >
>> >
>> >
>> >
>> > On Wed, Mar 27, 2013 at 5:47 AM, Anders Svensson <anders.otp@REDACTED>
>> > wrote:
>> >>
>> >> Hi Samuel.
>> >>
>> >> > Thanks for your reply. As my understanding also, the erlang diameter
>> >> > should
>> >> > not care how the other peers are implemented, as long as the peers
>> >> > follow
>> >> > the diameter specifications.
>> >> >
>> >> > But the erlang diameter library provides limited error information
>> >> > and
>> >> > this
>> >> > makes it harder to analyse problems, especially for newbies.
>> >>
>> >> It's usually (it seems) problems with capabilities exchange that trips
>> >> people up, and you're right that these can be a bit difficult to to
>> >> detect: capabilities exchange will fail, the transport connection will
>> >> be broken, and that's that. I'm a bit hesitant of logging ordinary
>> >> (albeit failed) Diameter signaling but I'll see what I can do.
>> >>
>> >> > My set-up is:
>> >> >
>> >> > - Client (192.168.193.58) runs from an erlang run time in a virtual
>> >> > machine.
>> >> > Just made some little changes based on the sample code:
>> >> >
>> >> >
>> >> > **********************************************************************************************************************
>> >> > -define(SVC_NAME, ?MODULE).
>> >> > -define(APP_RAR_ALIAS, rara).
>> >> > -define(APP_CCR_ALIAS, ccra).
>> >> > -define(DIAMETER_DICT_CCRA, diameter_gen_rfc4006_cc).
>> >> > -define(DIAMETER_APP_ID_CCRA, 16777238).
>> >> > -define(DIAMETER_DICT_COMMON_NEW, diameter_gen_base_rfc6733).
>> >> >
>> >> > -define(SERVICE(Name), [{'Origin-Host', ?L(Name) ++ ".example.com"},
>> >> >                         {'Origin-Realm', "example.com"},
>> >> >                         {'Vendor-Id', 193},
>> >> >                         {'Product-Name', "Client"},
>> >> >                         {'Auth-Application-Id',
>> >> > [?DIAMETER_APP_ID_COMMON, ?DIAMETER_APP_ID_CCRA]},
>> >> >                         {application, [{alias, ?APP_RAR_ALIAS},
>> >> >                                        {dictionary,
>> >> > ?DIAMETER_DICT_COMMON_NEW},
>> >> >                                        {module, client_cb}]},
>> >> >                         {application, [{alias, ?APP_CCR_ALIAS},
>> >> >                                        {dictionary,
>> >> > ?DIAMETER_DICT_CCRA},
>> >> >                                        {module, client_cb_ccra}]}]).
>> >> >
>> >> > connect(Name, T) ->
>> >> >     diameter:add_transport(Name, {connect, [{reconnect_timer,
>> >> > 5000000},
>> >> > {capx_timeout, 50000000} | client(T)]}).
>> >> >
>> >> > client(T) ->
>> >> >      client({T, {192,168,193,58}, {10,0,10,71}, 14302}).
>> >> >
>> >> >
>> >> > **********************************************************************************************************************
>> >>
>> >> So far, so good. (Although a 5000 sec reconnect_timer is on the long
>> >> side.
>> >> :)
>> >>
>> >> > - Server (10.0.10.71, port 14302) runs a diameter server remotely on
>> >> > a
>> >> > Linux. It is up and running and being tested with other diameter
>> >> > client.
>> >> >
>> >> > Do the followings:
>> >> >
>> >> > 2> debugger:start().
>> >> > {ok,<0.39.0>}
>> >> > 3> diameter:start().
>> >> > ok
>> >> > 4> client:start().
>> >> > ok
>> >> > 5> client:connect(tcp).
>> >> > {ok,#Ref<0.0.0.1210>}
>> >> > 6> client:call().
>> >> > {error,no_connection}
>> >> > 7>
>> >> >
>> >> > The print information from the runtime doesn't say too much, it shows
>> >> > no
>> >> > connection when call client:call(). In fact, when I check the
>> >> > debugger
>> >> > window, there is no tcp connection being set up properly when calling
>> >> > client:connect(tcp). Meaning the problem happened early without any
>> >> > reminder.
>> >> >
>> >> > It is also wired that the debugger monitor window doesn't allow to
>> >> > copy/paste:(, so I have to type some errors as following:
>> >> >
>> >> > <0.128.0> diameter_peer_fsm:init/1 exit
>> >> > {shutdown,{'DOWN',#Ref<0.0.0.1413>process,<0.136.0>, {shutdown,
>> >> > {tcp_closed,#Port<0.2250>}}}}
>> >> > <0.136.0> diameter_tcp:init/1 exit
>> >> > {shutdown,{tcp_closed,#Port<0.2250>}}
>> >>
>> >> This looks like the peer is closing the connection. We're not even
>> >> getting as far as sending CER. Are you able to see what's happening on
>> >> the peer at all, why it's closing the connection?
>> >>
>> >> /Anders, Erlang/OTP
>> >>
>> >>
>> >> > <0.138.0> diameter_tcp:init/1 exit {shutdown, {stop,<0.136.0>}}
>> >> >
>> >> > I increased the capx_timeout, since I thought might be CER timeout.
>> >> > No
>> >> > luck.
>> >> >
>> >> > There was no tcp connection being set up at all. I use the wireshark
>> >> > to
>> >> > watch on the interface with the filter <ip.src==192.168.193.58> and I
>> >> > didn't
>> >> > see any outgoing packets.
>> >> >
>> >> > Again, It still might be some configurations are incorrect causing
>> >> > the
>> >> > above
>> >> > problem. But with the limited information, it is difficult to figure
>> >> > out
>> >> > what was wrong for newbies.
>> >> >
>> >> > Could you provide some guidances on how to resolve this and any
>> >> > suggestions
>> >> > on diagnosing diameter or erlang issues?
>> >> >
>> >> > Thanks a lot for your comments!
>> >> >
>> >> > Samuel
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Mar 26, 2013 at 12:30 PM, Anders Svensson
>> >> > <anders.otp@REDACTED>
>> >> > wrote:
>> >> >>
>> >> >> On Tue, Mar 26, 2013 at 3:48 PM, S X <erlangprogram@REDACTED>
>> >> >> wrote:
>> >> >> > Hi, Rudolph & Anders,
>> >> >> >
>> >> >> > Not sure your problem is resolved or not.
>> >> >> >
>> >> >> > I was able to use the diameter sample code with the local/remote
>> >> >> > mode,
>> >> >> > i.e.
>> >> >> > Start a diameter server in an erts on a pc, and start a diameter
>> >> >> > client
>> >> >> > in
>> >> >> > an erts on another pc (I use virtual machines as pcs here). They
>> >> >> > can
>> >> >> > communicate different types of diameter messages without problems.
>> >> >> >
>> >> >> > However, when I try to use a diameter client (the sample code)
>> >> >> > from
>> >> >> > an
>> >> >> > erts
>> >> >> > to connect a diameter server which is not running within an erts
>> >> >> > (other
>> >> >> > diameter server not implemented in erlang). It doesn't work(can
>> >> >> > not
>> >> >> > build up
>> >> >> > a connection). I am not quite familiar with the detailed
>> >> >> > implementation
>> >> >> > of
>> >> >> > the erlang diameter library now. I am feeling that the erlang
>> >> >> > diameter
>> >> >> > relies on the erlang nodes, which means all the peers are built up
>> >> >> > only
>> >> >> > on
>> >> >> > the erts (distributed erlang runtimes). Does the erlang diameter
>> >> >> > only
>> >> >> > work
>> >> >> > within erlang environment? Or in order to talk to other diameter
>> >> >> > servers,
>> >> >>
>> >> >> No, diameter has no idea how any peers it connects to are
>> >> >> implemented.
>> >> >>
>> >> >> > does it need to write another erlang client using some functions
>> >> >> > from
>> >> >> > the
>> >> >> > erlang diameter library, like encode/decode? The sample client
>> >> >> > code
>> >> >> > doesn't
>> >> >> > work in this situation.
>> >> >> >
>> >> >> > I am not sure my understanding above is correct or not. Can you
>> >> >> > provide
>> >> >> > some
>> >> >> > guidance?
>> >> >>
>> >> >> Sure, if you provide me with an example of something that doesn't
>> >> >> work
>> >> >> as expected.
>> >> >>
>> >> >> Anders
>> >> >>
>> >> >>
>> >> >> > Thanks,
>> >> >> >
>> >> >> > Samuel
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Mar 21, 2013 at 9:00 AM, Anders Svensson
>> >> >> > <anders.otp@REDACTED>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> One more time to the list ...
>> >> >> >>
>> >> >> >> On Thu, Mar 21, 2013 at 12:58 PM, Anders Svensson
>> >> >> >> <anders.otp@REDACTED>
>> >> >> >> wrote:
>> >> >> >> > The problem looks to be that there's an {ssl, false} option
>> >> >> >> > being
>> >> >> >> > into
>> >> >> >> > diameter_tcp and then down to gen_tcp, which causes it to raise
>> >> >> >> > badarg. What OTP release is this? I can't say I recall the
>> >> >> >> > example
>> >> >> >> > code passing this tuple.
>> >> >> >> >
>> >> >> >> > /Anders, Erlang/OTP
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, Mar 20, 2013 at 6:08 PM, Rudolph van Graan
>> >> >> >> > <rvg.mailing@REDACTED>
>> >> >> >> > wrote:
>> >> >> >> >> Hi there,
>> >> >> >> >>
>> >> >> >> >> I'm trying to start up the Erlang diameter demo application
>> >> >> >> >> shipped
>> >> >> >> >> with
>> >> >> >> >> Erlang/OTP. The issue is that, no matter what format I try, I
>> >> >> >> >> can't
>> >> >> >> >> get
>> >> >> >> >> the
>> >> >> >> >> client to connect to a remote diameter server.
>> >> >> >> >>
>> >> >> >> >> In that example, I started the application as follows:
>> >> >> >> >>
>> >> >> >> >> 3>  diameter:start(), client:start().
>> >> >> >> >> ok
>> >> >> >> >>
>> >> >> >> >> 4> client:connect({tcp,{10,151,0,166},{10,249,20,174},3868}).
>> >> >> >> >> {ok,#Ref<0.0.0.643>}
>> >> >> >> >>
>> >> >> >> >> 7> client:call().
>> >> >> >> >> {error,no_connection}
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Here, my local IP address is {10,151,0,166} and the remote one
>> >> >> >> >> is
>> >> >> >> >> {10,249,20,174}.
>> >> >> >> >>
>> >> >> >> >> TCP to the server is working:
>> >> >> >> >>
>> >> >> >> >> telnet 10.249.20.174 3868
>> >> >> >> >> Trying 10.249.20.174...
>> >> >> >> >> Connected to 10.249.20.174.
>> >> >> >> >> Escape character is '^]'.
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I traced diameter_tcp and I can see that it is getting a
>> >> >> >> >> badarg
>> >> >> >> >> error
>> >> >> >> >> somewhere:
>> >> >> >> >>
>> >> >> >> >> (<0.114.0>) returned from diameter_tcp:start/3 ->
>> >> >> >> >> {ok,<0.115.0>,
>> >> >> >> >>
>> >> >> >> >> [{10,151,0,166}]}
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:handle_info({'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg},{monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:m({'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg},{monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) returned from diameter_tcp:m/2 -> ok
>> >> >> >> >>
>> >> >> >> >> Does anyone have an idea what I am doing wrong? My feeling is
>> >> >> >> >> that
>> >> >> >> >> it
>> >> >> >> >> has to
>> >> >> >> >> do with the local ip address. I don't understand why I even
>> >> >> >> >> need
>> >> >> >> >> to
>> >> >> >> >> supply a
>> >> >> >> >> local IP address and the documentation isn't very clear.
>> >> >> >> >>
>> >> >> >> >> Thanks,
>> >> >> >> >>
>> >> >> >> >> Rudolph
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>  Here is the trace:
>> >> >> >> >>
>> >> >> >> >> (<0.84.0>) call
>> >> >> >> >> diameter_tcp:start_link({monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >> diameter_tcp:init({monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) call diameter_tcp:i({monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) returned from diameter_tcp:i/1 ->
>> >> >> >> >> {monitor,<0.114.0>,<0.115.0>}
>> >> >> >> >> (<0.84.0>) returned from diameter_tcp:start_link/1 ->
>> >> >> >> >> {ok,<0.116.0>}
>> >> >> >> >> (<0.115.0>) call diameter_tcp:ssl([{ssl,false},
>> >> >> >> >>  {ip,{10,151,0,166}},
>> >> >> >> >>  {raddr,{10,249,20,174}},
>> >> >> >> >>  {rport,3868},
>> >> >> >> >>  {reuseaddr,true}])
>> >> >> >> >> (<0.115.0>) call diameter_tcp:ssl_opts([])
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:ssl_opts/1 -> false
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:ssl/1 -> {false,
>> >> >> >> >>                                                  [{ssl,false},
>> >> >> >> >>
>> >> >> >> >> {ip,{10,151,0,166}},
>> >> >> >> >>
>> >> >> >> >> {raddr,{10,249,20,174}},
>> >> >> >> >>
>> >> >> >> >> {rport,3868},
>> >> >> >> >>
>> >> >> >> >> {reuseaddr,true}]}
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:i(connect,#Ref<0.0.0.643>,gen_tcp,<0.114.0>,false,[{ssl,false},
>> >> >> >> >>  {ip,{10,151,0,166}},
>> >> >> >> >>  {raddr,{10,249,20,174}},
>> >> >> >> >>  {rport,3868},
>> >> >> >> >>  {reuseaddr,true}],[])
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:i(connect,#Ref<0.0.0.643>,gen_tcp,<0.114.0>,[{ssl,false},
>> >> >> >> >>  {ip,{10,151,0,166}},
>> >> >> >> >>  {raddr,{10,249,20,174}},
>> >> >> >> >>  {rport,3868},
>> >> >> >> >>  {reuseaddr,true}],[])
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >> diameter_tcp:get_addr([{ip,{10,151,0,166}}],[])
>> >> >> >> >> (<0.115.0>) call diameter_tcp:addr([{ip,{10,151,0,166}}],[])
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:addr/2 ->
>> >> >> >> >> {10,151,0,166}
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:get_addr/2 ->
>> >> >> >> >> {10,151,0,166}
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >> diameter_tcp:get_addr([{raddr,{10,249,20,174}}],[])
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >> diameter_tcp:addr([{raddr,{10,249,20,174}}],[])
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:addr/2 ->
>> >> >> >> >> {10,249,20,174}
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:get_addr/2 ->
>> >> >> >> >> {10,249,20,174}
>> >> >> >> >> (<0.115.0>) call diameter_tcp:get_port([{rport,3868}])
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:get_port/1 -> 3868
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:gen_opts({10,151,0,166},[{ssl,false},{reuseaddr,true}])
>> >> >> >> >> (<0.115.0>) returned from diameter_tcp:gen_opts/2 -> [binary,
>> >> >> >> >>
>> >> >> >> >> {packet,0},
>> >> >> >> >>
>> >> >> >> >> {active,once},
>> >> >> >> >>
>> >> >> >> >> {ip,{10,151,0,166}},
>> >> >> >> >>
>> >> >> >> >> {ssl,false},
>> >> >> >> >>
>> >> >> >> >> {reuseaddr,true}]
>> >> >> >> >> (<0.82.0>) returned from diameter_tcp:start_link/1 ->
>> >> >> >> >> {ok,<0.115.0>,
>> >> >> >> >>
>> >> >> >> >> [{10,151,0,166}]}
>> >> >> >> >> (<0.115.0>) call
>> >> >> >> >> diameter_tcp:connect(gen_tcp,{10,249,20,174},3868,[binary,
>> >> >> >> >>  {packet,0},
>> >> >> >> >>  {active,once},
>> >> >> >> >>  {ip,{10,151,0,166}},
>> >> >> >> >>  {ssl,false},
>> >> >> >> >>  {reuseaddr,true}])
>> >> >> >> >> (<0.114.0>) returned from diameter_tcp:start/3 ->
>> >> >> >> >> {ok,<0.115.0>,
>> >> >> >> >>
>> >> >> >> >> [{10,151,0,166}]}
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:handle_info({'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg},{monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:m({'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg},{monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) returned from diameter_tcp:m/2 -> ok
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:x({'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg})
>> >> >> >> >> (<0.116.0>) call
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> diameter_tcp:terminate({shutdown,{'DOWN',#Ref<0.0.0.924>,process,<0.115.0>,badarg}},{monitor,<0.114.0>,<0.115.0>})
>> >> >> >> >> (<0.116.0>) returned from diameter_tcp:terminate/2 -> ok
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> _______________________________________________
>> >> >> >> >> erlang-questions mailing list
>> >> >> >> >> erlang-questions@REDACTED
>> >> >> >> >> http://erlang.org/mailman/listinfo/erlang-questions
>> >> >> >> >>
>> >> >> >> _______________________________________________
>> >> >> >> erlang-questions mailing list
>> >> >> >> erlang-questions@REDACTED
>> >> >> >> http://erlang.org/mailman/listinfo/erlang-questions
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>



More information about the erlang-questions mailing list