<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hello,</div><div class=""><br class=""></div><div class="">This appears to be an interesting issue. First of all I’ve not seen a negative jitter and my impression was that your FSM loop suffer from measurement error. I’ve changed the measurement loop to be very-very tiny but there was not significant gain on jitter it is on par with your original code.</div><div class=""><br class=""></div><div class="">...</div><div class=""><div class="">T0 = os:timestamp(),</div><div class="">receive</div><div class="">...</div><div class="">after TickInterval -></div><div class=""> T1 = os:timestamp(),</div><div class=""> Jitter = timer:now_diff(T1, T0) - (TickInterval * 1000)</div><div class=""> ...</div><div class="">end</div><div class="">...</div></div><div class=""><br class=""></div><div class="">All-in-all, I’ve run the measurement both on my laptop Mac Book (Intel i5) and Amazon (cr1.8xlarge). The results was steady. You can find them below. I’ve seen jitter over 50ms for 10K process on my laptop only.</div><div class="">I am tend to thing that you are experience high jitter due to excessive CPU utilisation in your test bed. However, you can run the test with single FSM, the jitter is far from 0. May be it is time look VM internal. </div><div class=""><br class=""></div><div class="">You have proposed three possible solutions in your earlier email, I am afraid all of them will suffer from ‘jitter’ due to multiple reasons including the overhead in network communication. The option #1 (timer per fsm) still looks more feasible from my perspective. You might implement adaptive timer to minimise experienced errors. </div><div class=""><br class=""></div><div class="">Laptop:</div><div class="">kolesnik@pro:tmp$ erl -sbt ts -sws very_eager -swt high<br class="">Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]<br class=""><br class="">Eshell V6.2 (abort with ^G)<br class="">1> test_fsm5:go(1000, 50, 50, 1).<br class="">waiting for 1000 FSMs, tickrate 50<br class="">avg: 3164.69932160804<br class="">max: 13372<br class="">min: 4<br class="">median: 2426<br class="">95th: 8339<br class="">99th: 10552<br class="">all_done<br class=""><br class=""></div><div class="">AWS:</div><div class=""><br class=""></div><div class="">[ec2-user@xxx ~]$ /usr/local/xxx/erts-6.2/bin/erl -sbt ts -sws very_eager -swt high<br class="">Erlang/OTP 17 [erts-6.2] [source] [64-bit] [smp:32:32] [async-threads:10] [hipe] [kernel-poll:false]<br class=""><br class="">Eshell V6.2 (abort with ^G)<br class="">1> test_fsm5:go(1000, 50, 50, 1).<br class="">waiting for 1000 FSMs, tickrate 50<br class="">avg: 998.798351758794<br class="">max: 1926<br class="">min: 82<br class="">median: 998<br class="">95th: 1152<br class="">99th: 1204<br class="">all_done<br class=""><br class=""></div><div class="">- Dmitry</div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On 27 May 2015, at 00:03, Felix Gallo <<a href="mailto:felixgallo@gmail.com" class="">felixgallo@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Innovative thinking, Jesper! But in this case, in this testbed, the fsms aren't getting any messages other than those which they are delivering to themselves. Which adds to the intrigue. <div class=""><br class=""><div class="">I took your suggestion and tried using gen_fsm:start_timer/2. Interestingly it slightly increased the jitter variance and the negative jitter issue is still present. It's possible that my, ah, rapidly-and-pragmatically-built testbed suffers from some flaw, but I'm not seeing it.</div></div><div class=""><br class=""></div><div class="">Here's my code:</div><div class=""><br class=""></div><div class=""><a href="https://gist.github.com/anonymous/47cde5e60a619319053f" class="">https://gist.github.com/anonymous/47cde5e60a619319053f</a><br class=""></div><div class=""><br class=""></div><div class="">Here's sample output on this small but moderately modern non-cloud osx machine:</div><div class=""><br class=""></div><div class=""><div class="">> test_fsm5:go(1000,40,40,10).</div><div class="">waiting for 1000 FSMs, tickrate 40</div><div class="">avg: 1324.1012703862662</div><div class="">max: 50219</div><div class="">min: -184</div><div class="">median: 1018</div><div class="">95th: 2615</div><div class="">99th: 9698</div></div><div class=""><br class=""></div><div class="">note that the max is 50ms of jitter; the min is negative 184 us jitter, and the median jitter is about 1ms, which correlates well with my beliefs about scheduler wakeup timers...</div><div class=""><br class=""></div><div class="">F.</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, May 26, 2015 at 12:09 PM, Jesper Louis Andersen <span dir="ltr" class=""><<a href="mailto:jesper.louis.andersen@gmail.com" target="_blank" class="">jesper.louis.andersen@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, May 26, 2015 at 8:52 PM, Felix Gallo <span dir="ltr" class=""><<a href="mailto:felixgallo@gmail.com" target="_blank" class="">felixgallo@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="font-family: Courier, monospace; font-size: inherit; font-weight: bold;" class="">{next_state,NextStateName,NewStateData,Timeout}</span></blockquote></div><br class="">This explains why you sometimes get less than 30ms sleep times. If an event reaches the process before Timeout, then the timeout is not triggered. Also, it may explain the jitter you are seeing, because an early event will reset the timeout. Try using gen_fsm:start_timer/2 or erlang:send_after...</div><div class="gmail_extra"><br class=""></div><div class="gmail_extra">If the problem persists, check lcnt. If you are locked on the timer wheel, then consider release 18 :)<span class="HOEnZb"><font color="#888888" class=""><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="">J.</div>
</font></span></div></div>
</blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></body></html>