<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Max,<div><br></div><div>What is high load means? BTW, According to your CPU utilization you run a new test. What was a change to compare with previous test?</div><div><br></div><div>I would advice you to check this thread <a href="http://old.nabble.com/-BUG--linux-2.6.28-rc3-regression:-IRQ-smp_affinities-not-respected-td20336758.html">http://old.nabble.com/-BUG--linux-2.6.28-rc3-regression:-IRQ-smp_affinities-not-respected-td20336758.html</a></div><div>It is old but about your NIC</div><div><br></div><div><blockquote type="cite"><div>2. Why "Interrupts" are so high?</div></blockquote></div><div>I've been experience high interrupts only because of NIC driver...</div><div><br></div><div><blockquote type="cite"><div>1. Which values of IRQ and Interrupts are ok/normal? </div></blockquote><div>50 - 60K if I remember properly</div><br><blockquote type="cite"><div><br></div><div>2. Which values you see on your servers under high load?</div></blockquote></div><div><div>120K irq/sec ~12000 RPS and 1Gbpp bandwidth =)</div></div><div><br></div><div><blockquote type="cite"><div>3. Which OS is the best for high computation/traffic to host Erlang/OTP application?</div></blockquote></div><div>I am sorry to say but not Ubuntu =) I swap test env from Ubuntu to CentOS right after 10.04 due to issue with NIC drivers but that was a different story and has nothing to do with your symptoms. </div><div>Production is either CentOS or AWS default images</div><div><br></div><div>- Dmitry</div><div><br></div><div><div>On Jan 30, 2012, at 3:49 PM, Max Bourinov wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Hi Erlangers,<div><br></div><div>I see strange server behavior when I apply high load on my erlang node.</div><div><br></div><div>My OS is Ubuntu 11.04 64 bit.</div><div>I use Erlang/OTP exmpp with xpath.</div><div>I have ~3000 processes running. All processes are OTP.</div>
<div><br></div><div>My hardware is:</div><div><br></div><div><a href="http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1">http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1</a></div>
<div><br></div><div>CPU: 2 x Intel Xeon 5420 Quad Core 2.50 GHz</div><div>RAM 16 GB</div><div><br></div><div>My sysclt is the following:</div><div><br></div><div><div><font face="'courier new', monospace"><b> # General gigabit tuning: </b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.core.rmem_max = 16777216</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.core.wmem_max = 16777216</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_rmem = 10240 87380 16777216</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_wmem = 10240 65536 16777216</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_syncookies = 1</b></font></div><div><font face="'courier new', monospace"><b><br></b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.ip_local_port_range = 10000 65000</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.max_pid = 4000000</b></font></div>
<div><font face="'courier new', monospace"><b><br></b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_window_scaling = 1</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_timestamps = 1</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_sack = 1</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_no_metrics_save = 1</b></font></div><div><font face="'courier new', monospace"><b><br>
</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.sem = 250 32000 100 128</b></font></div><div><font face="'courier new', monospace"><b><br>
</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmmni = 4096</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>fs.file-max = 100000</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>vm.swappiness = 0</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>vm.vfs_cache_pressure = 50</b></font></div>
<div><font face="'courier new', monospace"><b><br></b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.msgmax = 665536</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmmax = 68719476736</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmall = 4294967296</b></font></div>
</div><div><br></div><div><br></div><div>Please see the end part of each graph.</div><div><br></div><div>This is CPU:</div><div><a href="http://img848.imageshack.us/img848/8999/cpuw.png">http://img848.imageshack.us/img848/8999/cpuw.png</a></div>
<div><br></div><div>This is IRQ:</div><div><a href="http://img7.imageshack.us/img7/259/irqstats.png">http://img7.imageshack.us/img7/259/irqstats.png</a></div><div><br></div><div>This is Interrupts:</div><div><a href="http://img851.imageshack.us/img851/1985/interrupts.png">http://img851.imageshack.us/img851/1985/interrupts.png</a></div>
<div><br></div><div>This is eth0:</div><div><a href="http://img259.imageshack.us/img259/5730/ifeth0.png">http://img259.imageshack.us/img259/5730/ifeth0.png</a></div><div><br></div><div><br></div><div>So, my questions about graphs are:</div>
<div><br></div><div>1. Why my IRQ are so high? Especially "Rescheduling interrupts"? I read somewhere that it could be because of power management and it is expectable that server consumes more energy under the load, but I have to idea how to fix it in this case.</div>
<div><br></div><div>2. Why "Interrupts" are so high?</div><div><br></div><div>Please, Erlangers, <b><u>those of you who has experience with highly loaded erlang nodes</u></b> tell me about your experience:</div>
<div><br></div><div>1. Which values of IRQ and Interrupts are ok/normal? </div><div><br></div><div>2. Which values you see on your servers under high load?</div><div><br></div><div>3. Which OS is the best for high computation/traffic to host Erlang/OTP application?</div>
<div><br></div><div>4. Which settings you apply to you OS (general description or configs are welcome) to manage high load on Erlang?</div><div><br></div><div><br></div><div>I hope that this information will be useful for all of us, because I believe everybody will face problems like I do one day when high load is applied.</div>
<div><br></div><div>Thank you all in advance!</div><div><br clear="all"><div>Best regards,</div><div>Max</div><br><br>
</div>
_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>http://erlang.org/mailman/listinfo/erlang-questions<br></blockquote></div><br></body></html>