<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><font face="Ubuntu">Thanks for the constructive feedback :)</font><br>
</p>
<div class="moz-cite-prefix">On 12/4/19 9:34 π.μ., Tristan Sloughter wrote:<br>
</div>
<blockquote type="cite" cite="mid:c3634aae-03ce-41f8-903c-61629f893209@www.fastmail.com">
<title></title>
<style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style>
<div>Yea, instrumentation from the beginning is a good bet. Shameless plug <a href="https://opencensus.io/quickstart/erlang/" moz-do-not-send="true">
https://opencensus.io/quickstart/erlang/</a> :) -- and prometheus.erl for vm metrics like Jesper suggests.<br>
</div>
<div><br>
</div>
<div>Tristan<br>
</div>
<div><br>
</div>
<div>On Fri, Apr 12, 2019, at 03:53, Jesper Louis Andersen wrote:<br>
</div>
<blockquote type="cite" id="qt">
<div dir="ltr">
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">My first recommendation is to add instrumentation to the system, so you can see what is going on:<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default"><br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">* Tristan already suggested looking at mailbox sizes<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">* Network blocking is worth investigating as well. Many small messages can lead to network overload situations<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">* Docker/Kubernetes environments tend to be noisy if a lot of work is running in them. In particular, if you have high-throughput systems banded with low latency systems, you are
going to run into trouble.<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">* Enable the Erlang system monitor. Get it to report on blocked ports and processes.<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">* Add VM metrics: prometheus for instance.<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default"><br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">The problem can be everywhere: Inside your code, the VM, docker, kernel, hardware, ... Your first goal is to narrow down that. Verify things are looking correct in each layer before
moving to the next.<br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default"><br>
</div>
<div style="font-family:arial, helvetica, sans-serif;" class="qt-gmail_default">The fact latency starts out at 1 second where we are at millisecond level locally, would suggest something has to do with the distribution. Either in your own code, or in the underlying
setup.<br>
</div>
</div>
<div><br>
</div>
<div class="qt-gmail_quote">
<div class="qt-gmail_attr" dir="ltr">On Thu, Apr 11, 2019 at 9:07 PM Konstantinos Kallas <<a href="mailto:konstantinos.kallas@hotmail.com" moz-do-not-send="true">konstantinos.kallas@hotmail.com</a>> wrote:<br>
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-color:rgb(204,
204,
204);border-left-style:solid;border-left-width:1px;padding-left:1ex;" class="qt-gmail_quote">
<div bgcolor="#FFFFFF">
<p><span style="font-family:Ubuntu" class="font">Hello,</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">I have an Erlang application where latency is crucial and a lot of small messages (tuples with an atom and integer) are exchanged between processes in different nodes.
</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">The main procedure is that a main process sends a small message to 4 worker processes in other Erlang nodes, the worker processes do some negligible processing, and then they reply back to the main node with
a small message. </span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">Each separate Erlang node is on a different docker container (generated from the erlang:21 docker image), and all the containers are connected using a standard docker bridge network.</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">I have noticed that latency (the time from when the first message is sent, and its replies arrive) linearly increases with time. It starts at 1 second and after 30 seconds of execution latency has become 10 seconds.</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">I have tried running all processes on the same erlang node, and then latency is (as expected) a couple milliseconds, so my assumption is that the problem could be caused by one (or more) of the following:</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">- Some misconfiguration of the Erlang nodes</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">- Some misconfiguration of the docker network/containers</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">- Some penalty imposed by the operating system/docker because a lot of small messages are exchanged</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">Has anyone encountered this issue, or does anyone know how to configure Erlang nodes (and the operating system) to reduce message latency?
</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">Thanks in advance.</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">Best,</span><br>
</p>
<p><span style="font-family:Ubuntu" class="font">Konstantinos</span><br>
</p>
</div>
<div>_______________________________________________<br>
</div>
<div>erlang-questions mailing list<br>
</div>
<div><a href="mailto:erlang-questions@erlang.org" moz-do-not-send="true">erlang-questions@erlang.org</a><br>
</div>
<div><a rel="noreferrer" href="http://erlang.org/mailman/listinfo/erlang-questions" moz-do-not-send="true">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div>
</blockquote>
</div>
<div><br>
</div>
<div><br>
</div>
<div>-- <br>
</div>
<div class="qt-gmail_signature" dir="ltr">J.<br>
</div>
<div>_______________________________________________<br>
</div>
<div>erlang-questions mailing list<br>
</div>
<div><a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
</div>
<div><a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</div>
<div><br>
</div>
</blockquote>
<div><br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
</body>
</html>