<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">My first recommendation is to add instrumentation to the system, so you can see what is going on:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">* Tristan already suggested looking at mailbox sizes</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">* Network blocking is worth investigating as well. Many small messages can lead to network overload situations</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">* Docker/Kubernetes environments tend to be noisy if a lot of work is running in them. In particular, if you have high-throughput systems banded with low latency systems, you are going to run into trouble.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">* Enable the Erlang system monitor. Get it to report on blocked ports and processes.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">* Add VM metrics: prometheus for instance.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The problem can be everywhere: Inside your code, the VM, docker, kernel, hardware, ... Your first goal is to narrow down that. Verify things are looking correct in each layer before moving to the next.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The fact latency starts out at 1 second where we are at millisecond level locally, would suggest something has to do with the distribution. Either in your own code, or in the underlying setup.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 11, 2019 at 9:07 PM Konstantinos Kallas <<a href="mailto:konstantinos.kallas@hotmail.com">konstantinos.kallas@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><font face="Ubuntu">Hello,</font></p>
<p><font face="Ubuntu">I have an Erlang application where latency is crucial and a lot of small messages (tuples with an atom and integer) are exchanged between processes in different nodes.
<br>
</font></p>
<p><font face="Ubuntu">The main procedure is that a main process sends a small message to 4 worker processes in other Erlang nodes, the worker processes do some negligible processing, and then they reply back to the main node with a small message.
<br>
</font></p>
<p><font face="Ubuntu">Each separate Erlang node is on a different docker container (generated from the erlang:21 docker image), and all the containers are connected using a standard docker bridge network.</font></p>
<p><font face="Ubuntu">I have noticed that latency (the time from when the first message is sent, and its replies arrive) linearly increases with time. It starts at 1 second and after 30 seconds of execution latency has become 10 seconds.</font></p>
<p><font face="Ubuntu">I have tried running all processes on the same erlang node, and then latency is (as expected) a couple milliseconds, so my assumption is that the problem could be caused by one (or more) of the following:</font></p>
<p><font face="Ubuntu">- Some misconfiguration of the Erlang nodes</font></p>
<p><font face="Ubuntu">- Some misconfiguration of the docker network/containers<br>
</font></p>
<p><font face="Ubuntu">- Some penalty imposed by the operating system/docker because a lot of small messages are exchanged</font></p>
<p><font face="Ubuntu">Has anyone encountered this issue, or does anyone know how to configure Erlang nodes (and the operating system) to reduce message latency?
<br>
</font></p>
<p><font face="Ubuntu">Thanks in advance.<br>
</font></p>
<p><font face="Ubuntu">Best,</font></p>
<p><font face="Ubuntu">Konstantinos<br>
</font></p>
</div>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" rel="noreferrer" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">J.</div>