Erlang DNS resolution and send operation speed when two nodes are splitted
Thu Nov 26 11:23:08 CET 2020
We have a productive system where one Erlang node connected to another node
performs badly when the second node is down. In such a case the message
the first node augment and the node does not manage to process its messages
quickly enough. The communication between the nodes is done with
When sending a message to the second node from the first node we use a call
With tcpdump, traces, trial and errors and reading some of the Erlang code,
have identified that the :inet_gethost_native.gethostbyname is invoked a
number of times as long as the second down is stopped. This increase in load
causes the function to sometimes take more than 3 seconds before returning a
value! With tcpdump I have measured how long the DNS queries take and can
that this increase in time is due to the Erlang code and not to the network
dns server (max time for a DNS query is 15ms).
I have created a proof of concept with a detailed README showing how to
reproduce the problem:
It creates much more IO and less CPU load than our production problem but
external behavior is similar: when the second node is restarted, the DNS
fast again. My knowledge of Erlang is limited so the code is in Elixir but
simple enough that everybody knowing Erlang should be able to reproduce the
problem in Erlang.
My team and I are surprised that such a situation put Erlang under stress
and that just disconnecting/stopping (with Ctrl-C or systemctl stop
<service>) the node causes this problem. After all, a node being down
should not be a big deal and DNS queries should be
We used Erlang 21, Elixir 1.10 and Debian Stretch when we discovered the
problem. We have know updated to Erlang 23, Elixir 1.11 and Debian Buster
the problem persists.
Would you consider this behavior a bug?
Do you have experience with such a situation?
I have posted a similar message on the elixir forum a few days ago but did
not get any answer so I think it's appropriate to ask here, some experts
may have more knowledge of the BEAM internals.
Any help will be highly appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions