epmd

[Richard Barrett]

Gordon Beaton wrote:
>On 26 May 2000 13:23:52 GMT, Richard Barrett wrote:
>  > Whenever I start an erlang process using erl -name ..., another copy
>  > of epmd is also created. This is regardless of whether an instance of
>  > epmd is already running as a a result of my having run it explicitly
>  > from the command line or previously started another instance of erl.
>  > I've checked using netstat and lsof and all these instances of epmd
>  > are trying to listen on port 4369.
>
>This should not be possible! Epmd will exit at start if its attempt to
>bind the socket to port 4369 fails. And if another process (such as an
>earlier epmd) has already bound to the same port, then the attempt
>*will* fail, unless something is seriously wrong with your kernel.
>
>Are you quite certain that you have two running at the same time, on
>the same host? Please show the output of ps!
>
>Note that each of the erlang nodes maintains a connection to epmd, and
>netstat will show this connection with port 4369 at the "foreign" end.
>
>/gordon
>
>--
>g o r d o n . b e a t o n @ e r i c s s o n . c o m

I also wondered if I was delusional but the following output has 
convinced me I am not.

I am still investigating why this is occuring.

One line of enquiry a colleague has suggested to me is that if the 
SO_REUSEADDR socket option is specified then more than one bind to 
port 4369 could succeed (see TCP/IP Illustrated Vol 1 by W. Richard 
Stevens, Section 18.6 under Heading 2MSL Wait State, page 243 in my 
copy). If this hypothesis is correct then simply using bind failure 
to determine that a socket is not already in use may be insufficient 
to test for an existing epmd daemon, on Suse 6.2 Linux at least.

Incidentally, the machine concerned is running various other servers 
without any obvious problems although it is clear that my use of 
erlang is challenging the OS support for sockets in new and 
interesting ways. btw, I am using a stock kernel off the installation 
CDs rather than some home crafted version of my own, so the problem 
may be generic with erlang under Suse 6.2 Linux.


-----------------------------------------------------------

grep to find no copies of erl and epmd running:

vvvvvvvv
barrett@REDACTED:~ > ps ax | grep erl
19435  ?  S    0:00 grep erl
^^^^^^^^
-----------------------------------------------------------
Run one copy of erl as follows:

vvvvvvvv
barrett@REDACTED:~/erlang > erl -name fred
Erlang (BEAM) emulator version 4.9.1 [source]

Eshell V4.9.1  (abort with ^G)
(fred@REDACTED)1>
^^^^^^^^

grep again. we have a copy of beam and epmd running as expected:
vvvvvvvv
barrett@REDACTED:~ > ps ax | grep erl
19436  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/beam -- 
-root /usr/loca
19442  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/epmd -daemon
^^^^^^^^
and use epmd to find out who is about:

vvvvvvvv
barrett@REDACTED:~ > epmd -names
bash: epmd: command not found
barrett@REDACTED:~ > /usr/local/lib/erlang/erts-4.9.1/bin/epmd -names
epmd: up and running on port 4369 with data:
name fred at port 1752
^^^^^^^^
-----------------------------------------------------------
Second copy of erl run as follows:

vvvvvvvv
barrett@REDACTED:~/erlang > erl -name jim
Erlang (BEAM) emulator version 4.9.1 [source]

Eshell V4.9.1  (abort with ^G)
(jim@REDACTED)1>
^^^^^^^^
grep again. lo and behold, we now have two copies of epmd running on 
the same machine:

vvvvvvvv
barrett@REDACTED:~ > ps ax | grep erl
19436  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/beam -- 
-root /usr/loca
19442  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/epmd -daemon
19447  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/beam -- 
-root /usr/loca
19453  ?  S    0:00 /usr/local/lib/erlang/erts-4.9.1/bin/epmd -daemon
^^^^^^^^
and use epmd to find out who is about. only the most recently run 
beam is being reported:

vvvvvvvv
barrett@REDACTED:~ > /usr/local/lib/erlang/erts-4.9.1/bin/epmd -names
epmd: up and running on port 4369 with data:
name jim at port 1755
^^^^^^^^
use netstat to find out about which of the processes of interest is 
dealing with which ports. note the "impossible": both copies of epmd 
are bound to port 4369!

vvvvvvvv
barrett@REDACTED:~ > netstat -p | grep epmd
(Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
tcp        0      0 localhost.ftel.co.:4369 localhost.ftel.co.:1756 
ESTABLISHED 19453/epmd
tcp        0      0 localhost.ftel.co.:4369 localhost.ftel.co.:1753 
ESTABLISHED 19442/epmd
barrett@REDACTED:~ > netstat -p | grep beam
(Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
tcp        0      0 localhost.ftel.co.:1756 localhost.ftel.co.:4369 
ESTABLISHED 19447/beam
tcp        0      0 localhost.ftel.co.:1753 localhost.ftel.co.:4369 
ESTABLISHED 19436/beam
^^^^^^^^
check with lsof to see what it says about the open files for the epmd 
and beam processes. note the "impossible" again: both copies of epmd 
are listening on port 4369!

vvvvvvvv
barrett@REDACTED:~ > lsof -p 19436,19442,19447,19453 | grep TCP
beam    19436 barrett    3u  inet 264273            TCP *:1752 (LISTEN)
beam    19436 barrett    4u  inet 264275            TCP 
localhost.ftel.co.uk:1753->localhost.ftel.co.uk:4369 (ESTABLISHED)
epmd    19442 barrett    3u  inet 264271            TCP *:4369 (LISTEN)
epmd    19442 barrett    4u  inet 264276            TCP 
localhost.ftel.co.uk:4369->localhost.ftel.co.uk:1753 (ESTABLISHED)
beam    19447 barrett    3u  inet 264298            TCP *:1755 (LISTEN)
beam    19447 barrett    4u  inet 264300            TCP 
localhost.ftel.co.uk:1756->localhost.ftel.co.uk:4369 (ESTABLISHED)
epmd    19453 barrett    3u  inet 264296            TCP *:4369 (LISTEN)
epmd    19453 barrett    4u  inet 264301            TCP 
localhost.ftel.co.uk:4369->localhost.ftel.co.uk:1756 (ESTABLISHED)
^^^^^^^^


------------------------------------------------------------------
Richard Barrett, PostPoint 27,         e-mail:r.barrett@REDACTED
Fujitsu Telecommunications Europe Ltd,      tel: (44) 121 717 6337
Solihull Parkway, Birmingham Business Park, B37 7YU, England
"Democracy is two wolves and a lamb voting on what to have for
lunch. Liberty is a well armed lamb contesting the vote."
Benjamin Franklin, 1759
------------------------------------------------------------------