[erlang-questions] High latency when exchanging small messages between different Erlang nodes

[Konstantinos Kallas]

Hello,

I have an Erlang application where latency is crucial and a lot of small messages (tuples with an atom and integer) are exchanged between processes in different nodes.

The main procedure is that a main process sends a small message to 4 worker processes in other Erlang nodes, the worker processes do some negligible processing, and then they reply back to the main node with a small message.

Each separate Erlang node is on a different docker container (generated from the erlang:21 docker image), and all the containers are connected using a standard docker bridge network.

I have noticed that latency (the time from when the first message is sent, and its replies arrive) linearly increases with time. It starts at 1 second and after 30 seconds of execution latency has become 10 seconds.

I have tried running all processes on the same erlang node, and then latency is (as expected) a couple milliseconds, so my assumption is that the problem could be caused by one (or more) of the following:

- Some misconfiguration of the Erlang nodes

- Some misconfiguration of the docker network/containers

- Some penalty imposed by the operating system/docker because a lot of small messages are exchanged

Has anyone encountered this issue, or does anyone know how to configure Erlang nodes (and the operating system) to reduce message latency?

Thanks in advance.

Best,

Konstantinos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190411/8248e396/attachment.htm>