Hi Erlangers,<div><br></div><div>I see strange server behavior when I apply high load on my erlang node.</div><div><br></div><div>My OS is Ubuntu 11.04 64 bit.</div><div>I use Erlang/OTP exmpp with xpath.</div><div>I have ~3000 processes running. All processes are OTP.</div>
<div><br></div><div>My hardware is:</div><div><br></div><div><a href="http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1">http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1</a></div>
<div><br></div><div>CPU: 2 x Intel Xeon 5420 Quad Core 2.50 GHz</div><div>RAM 16 GB</div><div><br></div><div>My sysclt is the following:</div><div><br></div><div><div><font face="'courier new', monospace"><b> # General gigabit tuning: </b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.core.rmem_max = 16777216</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.core.wmem_max = 16777216</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_rmem = 10240 87380 16777216</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_wmem = 10240 65536 16777216</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_syncookies = 1</b></font></div><div><font face="'courier new', monospace"><b><br></b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.ip_local_port_range = 10000 65000</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.max_pid = 4000000</b></font></div>
<div><font face="'courier new', monospace"><b><br></b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_window_scaling = 1</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_timestamps = 1</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_sack = 1</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>net.ipv4.tcp_no_metrics_save = 1</b></font></div><div><font face="'courier new', monospace"><b><br>
</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.sem = 250 32000 100 128</b></font></div><div><font face="'courier new', monospace"><b><br>
</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmmni = 4096</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>fs.file-max = 100000</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>vm.swappiness = 0</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>vm.vfs_cache_pressure = 50</b></font></div>
<div><font face="'courier new', monospace"><b><br></b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.msgmax = 665536</b></font></div>
<div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmmax = 68719476736</b></font></div><div><b style="font-family:'courier new',monospace"> </b><font face="'courier new', monospace"><b>kernel.shmall = 4294967296</b></font></div>
</div><div><br></div><div><br></div><div>Please see the end part of each graph.</div><div><br></div><div>This is CPU:</div><div><a href="http://img848.imageshack.us/img848/8999/cpuw.png">http://img848.imageshack.us/img848/8999/cpuw.png</a></div>
<div><br></div><div>This is IRQ:</div><div><a href="http://img7.imageshack.us/img7/259/irqstats.png">http://img7.imageshack.us/img7/259/irqstats.png</a></div><div><br></div><div>This is Interrupts:</div><div><a href="http://img851.imageshack.us/img851/1985/interrupts.png">http://img851.imageshack.us/img851/1985/interrupts.png</a></div>
<div><br></div><div>This is eth0:</div><div><a href="http://img259.imageshack.us/img259/5730/ifeth0.png">http://img259.imageshack.us/img259/5730/ifeth0.png</a></div><div><br></div><div><br></div><div>So, my questions about graphs are:</div>
<div><br></div><div>1. Why my IRQ are so high? Especially "Rescheduling interrupts"? I read somewhere that it could be because of power management and it is expectable that server consumes more energy under the load, but I have to idea how to fix it in this case.</div>
<div><br></div><div>2. Why "Interrupts" are so high?</div><div><br></div><div>Please, Erlangers, <b><u>those of you who has experience with highly loaded erlang nodes</u></b> tell me about your experience:</div>
<div><br></div><div>1. Which values of IRQ and Interrupts are ok/normal? </div><div><br></div><div>2. Which values you see on your servers under high load?</div><div><br></div><div>3. Which OS is the best for high computation/traffic to host Erlang/OTP application?</div>
<div><br></div><div>4. Which settings you apply to you OS (general description or configs are welcome) to manage high load on Erlang?</div><div><br></div><div><br></div><div>I hope that this information will be useful for all of us, because I believe everybody will face problems like I do one day when high load is applied.</div>
<div><br></div><div>Thank you all in advance!</div><div><br clear="all"><div>Best regards,</div><div>Max</div><br><br>
</div>