<div><div dir="auto">Hi Michael</div></div><div dir="auto"><br></div><div dir="auto">All packets in transit have a “seq_id” (sequential number).</div><div dir="auto"><br></div><div dir="auto">This means that in theory packet1, packet2...packetN can be checked in parallel and in any order (which is not the case in my current design), but they must be send to the next processing stage in order: packet1 first, then packet2...</div><div dir="auto"><br></div><div dir="auto">I would love to hear from you how can I turn this long-lived process to multiple short-lived ones while enforcing ordering.</div><div dir="auto"><br></div><div dir="auto">/Frank</div><div><div class="gmail_quote"><div dir="ltr"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"></div><div text="#000000" bgcolor="#FFFFFF">
<div class="m_-4386278282798874613moz-cite-prefix">On 2/1/19 8:11 AM, Frank Muller wrote:<br>
</div>
</div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"></blockquote></div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">I
tried two solutions to reduce the memory usage of the
problematic process:</span><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">1.
calling garbage:collect/0 after processing N packets (varying
N=10..128).</span><br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">Nothing
changed at all and the bin_alloc memory stayed fragmented as you
can see:</span><br style="color:rgb(49,49,49);word-spacing:1px">
<a href="http://147.135.36.188:3400/observer_cli_BEFORE.jpg" rel="noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://147.135.36.188:3400/observer_cli_BEFORE.jpg</a><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">The
call to instrument:carriers/0:</span><br style="color:rgb(49,49,49);word-spacing:1px">
<a href="http://147.135.36.188:3400/instrument_carriers.jpg" rel="noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://147.135.36.188:3400/instrument_carriers.jpg</a><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">The
call to instrument:allocations/0:</span><br style="color:rgb(49,49,49);word-spacing:1px">
<a href="http://147.135.36.188:3400/instrument_allocations.jpg" rel="noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://147.135.36.188:3400/instrument_allocations.jpg</a><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">2.
Hibernating the process after processing N packets (varying
N=10..128).</span><br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">The
HIT rates went above 90% immediately.</span><br style="color:rgb(49,49,49);word-spacing:1px">
<a href="http://147.135.36.188:3400/observer_cli_AFTER.jpg" rel="noreferrer" style="font-size:1rem;word-spacing:1px" target="_blank">http://147.135.36.188:3400/observer_cli_AFTER.jpg</a><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">What
is the effect of hibernating this process on the long term? </span><br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">This
process is receiving about ~1200 packets/sec under normal load
and can reach ~3000 packets/sec under heavy load.</span><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">Is
there a better way of solving the problem by tweeting the bin
allocator SBC/MBC?</span><br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<br style="color:rgb(49,49,49);word-spacing:1px">
<span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">/Frank</span>
<br>
<fieldset class="m_-4386278282798874613mimeAttachmentHeader"></fieldset>
</blockquote></div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><pre class="m_-4386278282798874613moz-quote-pre">_______________________________________________
erlang-questions mailing list
<a class="m_-4386278282798874613moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>
<a class="m_-4386278282798874613moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<p><tt>If you move the creation of temporary binaries out of any
Erlang processes you have that are long-lived, into short-lived
Erlang processes, you would no longer have this problem. The
tuning discussions, allocator options, hibernate use, etc. is
not solving the cause of the problem. Source code should not
need to call garbage:collect/0 and using temporary Erlang
processes makes the garbage collection occur naturally, at a
pace that shouldn't require special tuning.</tt></p>
<tt>Best Regards,</tt><br>
<tt>Michael<br>
</tt>
</div>
</blockquote></div></div>