<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Knut, <div><br></div><div>have you tried to play with scheduler wakeup thresholds?</div><div>I would probably try running beam with '+swt very_low'</div><div>(see <a href="http://erlang.org/doc/man/erl.html">http://erlang.org/doc/man/erl.html</a> for details). </div><div><br></div><div>Best regards,</div><div>Michal Ptaszek</div><div><br></div><div><br><div><div>On Feb 16, 2012, at 5:00 PM, Knut Nesheim wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi,<br><br>We just tried moving more load to the node using only two schedulers.<br>The VM is using 100% cpu on the two cores, with a run queue hovering<br>between 50 and 100 processes. There is no change in active schedulers.<br><br>As far as I can see, the load is now at a point where the schedulers<br>should become active. Is there anything we can do to understand more<br>about why this is not happening?<br><br>Knut<br><br>On Thu, Feb 16, 2012 at 10:19 AM, Knut Nesheim <<a href="mailto:knutin@gmail.com">knutin@gmail.com</a>> wrote:<br><blockquote type="cite">Thanks for a quick reply.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">During the GMT night which is our low point, we see quite low traffic<br></blockquote><blockquote type="cite">on each machine with maybe 5% cpu utilization of each of the 8 cores.<br></blockquote><blockquote type="cite">This could explain why some schedulers are put as idle. When traffic<br></blockquote><blockquote type="cite">picks up, we're using around 25% of each core.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The problem for us is that when traffic picks up again, the VM did not<br></blockquote><blockquote type="cite">start using the idle schedulers again. erlang:statistics(run_queue)<br></blockquote><blockquote type="cite">reports that there almost always are processes waiting, with peaks of<br></blockquote><blockquote type="cite">around 20 processes in our case. This causes much higher latency than<br></blockquote><blockquote type="cite">when using all cores.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We will try pushing more load to this node to see if it starts using<br></blockquote><blockquote type="cite">the idle schedulers.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Thanks<br></blockquote><blockquote type="cite">Knut<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Thu, Feb 16, 2012 at 8:01 AM, Kenneth Lundin<br></blockquote><blockquote type="cite"><<a href="mailto:kenneth.lundin@gmail.com">kenneth.lundin@gmail.com</a>> wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">What is your real problem except that there is no reported cpu load on some<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">of the cores?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">When there is low load on the system i.e. when there are too few runnable<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Erlang processes some of the schedulers will automatically be put as idle<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">letting the remaining schedulers run all processes. When the load increases<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the idle schedulers will be taken into service again.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Each scheduler normally runs on one processor core.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Can this explain what you have observed?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">/Kenneth Erlang/OTP Ericsson<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Den 15 feb 2012 17:53 skrev "Knut Nesheim" <<a href="mailto:knutin@gmail.com">knutin@gmail.com</a>>:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Dear list,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">From time to time we see the VM using only some of the available<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">cores. Typically 2 or 4 out of 8 available cores will be busy, while<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">the others are not used at all. From 'top' and 'htop' we see only some<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">cores being used.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">We are running the halfword emulator on 8 core machines and the VM is<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">correctly reporting the number of schedulers and the underlying<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">topology. The nodes are utilizing roughly 25% of each of the 8 cores<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">during normal operation.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">This problem appeared again today and we still have the process<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">running if anyone has any ideas for how to debug the issue.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">We are on Ubuntu 10.04 LTS, 2.6.32-342-ec2, 64bit, R14B04 with the<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">halfword emulator. The only NIF library we are using is jiffy.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Regards<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Knut<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">erlang-questions mailing list<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></blockquote></blockquote>_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>http://erlang.org/mailman/listinfo/erlang-questions<br></div></blockquote></div><br></div></body></html>