[erlang-questions] Strange server behavior when high load on erlang node

[Max Bourinov]

Hi Erlangers,

I see strange server behavior when I apply high load on my erlang node.

My OS is Ubuntu 11.04 64 bit.
I use Erlang/OTP exmpp with xpath.
I have ~3000 processes running. All processes are OTP.

My hardware is:

http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1

CPU: 2 x Intel Xeon 5420 Quad Core 2.50 GHz
RAM 16 GB

My sysclt is the following:

*    # General gigabit tuning:
                                                                     *
*    **net.core.rmem_max = 16777216*
*    **net.core.wmem_max = 16777216*
*    **net.ipv4.tcp_rmem = 10240 87380 16777216*
*    **net.ipv4.tcp_wmem = 10240 65536 16777216*
*    **net.ipv4.tcp_syncookies = 1*
*
*
*    **net.ipv4.ip_local_port_range = 10000   65000*
*    **kernel.max_pid = 4000000*
*
*
*    **net.ipv4.tcp_window_scaling = 1*
*    **net.ipv4.tcp_timestamps = 1*
*    **net.ipv4.tcp_sack = 1*
*    **net.ipv4.tcp_no_metrics_save = 1*
*
*
*    **kernel.sem = 250 32000 100 128*
*
*
*    **kernel.shmmni = 4096*
*    **fs.file-max = 100000*
*    **vm.swappiness = 0*
*    **vm.vfs_cache_pressure = 50*
*
*
*    **kernel.msgmax = 665536*
*    **kernel.shmmax = 68719476736*
*    **kernel.shmall = 4294967296*


Please see the end part of each graph.

This is CPU:
http://img848.imageshack.us/img848/8999/cpuw.png

This is IRQ:
http://img7.imageshack.us/img7/259/irqstats.png

This is Interrupts:
http://img851.imageshack.us/img851/1985/interrupts.png

This is eth0:
http://img259.imageshack.us/img259/5730/ifeth0.png


So, my questions about graphs are:

1. Why my IRQ are so high? Especially "Rescheduling interrupts"? I read
somewhere that it could be because of power management and it is expectable
that server consumes more energy under the load, but I have to idea how to
fix it in this case.

2. Why "Interrupts" are so high?

Please, Erlangers, *those of you who has experience with highly loaded
erlang nodes* tell me about your experience:

1. Which values of IRQ and Interrupts are ok/normal?

2. Which values you see on your servers under high load?

3. Which OS is the best for high computation/traffic to host Erlang/OTP
application?

4. Which settings you apply to you OS (general description or configs are
welcome) to manage high load on Erlang?


I hope that this information will be useful for all of us, because I
believe everybody will face problems like I do one day when high load is
applied.

Thank you all in advance!

Best regards,
Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120130/c89177d5/attachment.htm>