[erlang-questions] epmd regression bug in erlang-solutions.com esl-erlang 18.3?

Mon Mar 21 12:46:04 CET 2016

On Mon, Mar 21, 2016 at 05:57:45PM +0800, Khitai Pang wrote:
> On 2016/3/21 5:27, Michael Santos wrote:
> >On Mon, Mar 21, 2016 at 02:23:05AM +0800, Khitai Pang wrote:
> >>On 2016/3/20 6:53, Michael Santos wrote:
> >>>On Sat, Mar 19, 2016 at 11:47:49PM +0800, Khitai Pang wrote:
> >>>>On 2016/3/19 19:45, Michael Santos wrote:
> >>>>>On Mar 18, 2016 10:46 PM, "Khitai Pang" <khitai.pang@REDACTED
> >>>>><mailto:khitai.pang@REDACTED>> wrote:
> >>>>>>Hi,
> >>>>>>
> >>>>>>I have been using the esl-erlang pacakge from
> >>>>>https://packages.erlang-solutions.com/erlang/ Ubuntu PPA for
> >>>>>Trusty.
> >>>>>>I have these in my bashrc:
> >>>>>>
> >>>>>>export ERL_EPMD_ADDRESS="XXX.XXX.XXX.XXX"   (LAN IP of my VM in
> >>>>>the cloud)
> >>>>>>export ERL_EPMD_PORT="5678"
> >>>>>Just tested and it worked for me.
> >>>>>
> >>>>>Does the IP address in ERL_EPMD_ADDRESS exist in the VM?
> >>>>>
> >>>>Yes it does, it's the IP of the LAN in the clound, and it all works
> >>>>fine with 18.2.
> >>>>
> >>>>I believe it's possible that the issue is caused by the
> >>>>configuration in the operating system (Ubuntu 14.04.4 x86_64) but
> >>>>currently I have no idea how to further debug this issue.
> >>>>
> >>>>With esl-erlang 18.2 and the following configs in bashrc:
> >>>>
> >>>>export ERL_EPMD_ADDRESS="XXX.XXX.XXX.XXX"
> >>>>export ERL_EPMD_PORT="5678"
> >>>>
> >>>>epmd works fine:
> >>>>
> >>>>tcp        0      0 127.0.0.1:5678          0.0.0.0:* LISTEN      6754/epmd
> >>>>tcp        0      0 10.131.70.80:5678       0.0.0.0:* LISTEN      6754/epmd
> >>>>
> >>>>
> >>>>Thanks
> >>>>Khitai
> >>>>
> >>>>>>Everything worked fine with the 18.2 package.  After upgrading
> >>>>>to esl-erlang 18.3, epmd stopped to work:
> >>>>>>>epmd -debug
> >>>>>>epmd: Sat Mar 19 16:16:51 2016: epmd running - daemon = 0
> >>>>>>epmd: Sat Mar 19 16:16:51 2016: failed to bind socket: Cannot
> >>>>>assign requested address
> >>>>>>There is a workaround: if I remove the ERL_EPMD_ADDRESS
> >>>>>environment variable, epms works:
> >>>>>>tcp        0      0 0.0.0.0:5678 <http://0.0.0.0:5678> 0.0.0.0:*
> >>>>>LISTEN      3328/epmd
> >>>>>>tcp6       0      0 :::5678 :::*                    LISTEN     3328/epmd
> >>>>>>After downgrading esl-erlang from 18.3 to 18.2, the issue is gone.
> >>>Have you disabled IPv6?
> >>>
> >>>     # disable IPv6 support on the loopback device
> >>>     $ sudo sysctl net.ipv6.conf.lo.disable_ipv6=1
> >>>
> >>>     $ ERL_EPMD_ADDRESS="127.0.0.1" ERL_EPMD_PORT=5678 epmd
> >>>     epmd: Sat Mar 19 22:40:09 2016: failed to bind socket: Cannot assign requested address
> >>>
> >>>     # enable IPv6 support on the loopback device
> >>>     $ sudo sysctl net.ipv6.conf.lo.disable_ipv6=0
> >>>
> >>>     $ ERL_EPMD_ADDRESS="127.0.0.1" ERL_EPMD_PORT=5678 epmd
> >>>     <works>
> >>>
> >>>If we look at what is happening:
> >>>
> >>>     $ ERL_EPMD_ADDRESS="127.0.0.1" ERL_EPMD_PORT=5678 strace epmd
> >>>     ...
> >>>     bind(4, {sa_family=AF_INET6, sin6_port=htons(5678), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
> >>>     ...
> >>>
> >>>When you specify a specific address to bind, epmd will also bind to the
> >>>IPv4 and IPv6 loopbacks. In this case, your OS supports IPv6 but has
> >>>disabled it on the loopback device so the bind fails:
> >>>
> >>>     $ ifconfig lo
> >>>     lo        Link encap:Local Loopback
> >>>               inet addr:127.0.0.1  Mask:255.0.0.0
> >>>               UP LOOPBACK RUNNING  MTU:65536  Metric:1
> >>>               RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >>>               TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> >>>               collisions:0 txqueuelen:0
> >>>               RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
> >>>
> >>>So either enable IPv6 for the loopback or disable IPv6 completely by
> >>>passing "ipv6.disable=1" at boot.
> >>Michael,
> >>
> >>Thanks for the workaround.  2 questions:
> >>
> >>1)  I have these in my sysctl:
> >>
> >>net.ipv6.conf.all.disable_ipv6 = 1
> >>net.ipv6.conf.default.disable_ipv6 = 1
> >>net.ipv6.conf.eth0.disable_ipv6 = 1
> >>net.ipv6.conf.eth1.disable_ipv6 = 1
> >>net.ipv6.conf.lo.disable_ipv6 = 1
> >>
> >>Aren't these enough to tell epmd that ipv6 is not available on loopback?
> >We can see from your first email that allocating an IPv6 socket and
> >binding it to the IPv6 ANY address works:
> >
> >tcp        0      0 0.0.0.0:5678 0.0.0.0:* LISTEN      3328/epmd
> >tcp6       0      0 :::5678      :::*      LISTEN      3328/epmd
> >
> >A system with a working IPv6 implementation must have an IPv6 loopback.
> >So I would say your system configuration is broken. If the intent was to
> >disable IPv6, it didn't work: you have a half-working system.
> 
> Agreed.
> 
> >I do think epmd should require explicitly specifying all addresses to
> >bind instead of implicitly binding the loopback. So in your case you
> >would need to add the IPv4 loopback but it would otherwise work as
> >before:
> >
> >     $ ERL_EPMD_ADDRESS="10.131.70.80,127.0.0.1" ERL_EPMD_PORT=5678 epmd
> 
> So I should either disable IPv6 completely or enable IPv6 on loopback?

Yes, that's correct:

    $ sudo sysctl net.ipv6.conf.lo.disable_ipv6=0

Or:

    # /etc/default/grub
    # update-grub
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash ipv6.disable=1"

> >>2)  Why does 18.2 work in the same environment?
> >18.3 supports node registration for distribution over IPv6 (OTP-13364):
> >
> >http://erlang.org/download/otp_src_18.3.readme
> 
> I see, thanks for the information.
> 
> 
> Thanks
> Khitai