[erlang-questions] Strange server behavior when high load on erlang node

[Dmitry Kolesnikov]

Max,

What is high load means? BTW, According to your CPU utilization you run a new test. What was a change to compare with previous test?

I would advice you to check this thread http://old.nabble.com/-BUG--linux-2.6.28-rc3-regression:-IRQ-smp_affinities-not-respected-td20336758.html
It is old but about your NIC

> 2. Why "Interrupts" are so high?

I've been experience high interrupts only because of NIC driver...

> 1. Which values of IRQ and Interrupts are ok/normal? 
50 - 60K if I remember properly

> 
> 2. Which values you see on your servers under high load?

120K irq/sec ~12000 RPS and 1Gbpp bandwidth =)

> 3. Which OS is the best for high computation/traffic to host Erlang/OTP application?

I am sorry to say but not Ubuntu =) I swap test env from Ubuntu to CentOS right after 10.04 due to issue with NIC drivers but that was a different story and has nothing to do with your symptoms. 
Production is either CentOS or AWS default images

- Dmitry

On Jan 30, 2012, at 3:49 PM, Max Bourinov wrote:

> Hi Erlangers,
> 
> I see strange server behavior when I apply high load on my erlang node.
> 
> My OS is Ubuntu 11.04 64 bit.
> I use Erlang/OTP exmpp with xpath.
> I have ~3000 processes running. All processes are OTP.
> 
> My hardware is:
> 
> http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/3709945-3709945-3328410-241641-3328419-1842750.html?dnr=1
> 
> CPU: 2 x Intel Xeon 5420 Quad Core 2.50 GHz
> RAM 16 GB
> 
> My sysclt is the following:
> 
>     # General gigabit tuning:                                                                                                                 
>     net.core.rmem_max = 16777216
>     net.core.wmem_max = 16777216
>     net.ipv4.tcp_rmem = 10240 87380 16777216
>     net.ipv4.tcp_wmem = 10240 65536 16777216
>     net.ipv4.tcp_syncookies = 1
> 
>     net.ipv4.ip_local_port_range = 10000   65000
>     kernel.max_pid = 4000000
> 
>     net.ipv4.tcp_window_scaling = 1
>     net.ipv4.tcp_timestamps = 1
>     net.ipv4.tcp_sack = 1
>     net.ipv4.tcp_no_metrics_save = 1
> 
>     kernel.sem = 250 32000 100 128
> 
>     kernel.shmmni = 4096
>     fs.file-max = 100000
>     vm.swappiness = 0
>     vm.vfs_cache_pressure = 50
> 
>     kernel.msgmax = 665536
>     kernel.shmmax = 68719476736
>     kernel.shmall = 4294967296
> 
> 
> Please see the end part of each graph.
> 
> This is CPU:
> http://img848.imageshack.us/img848/8999/cpuw.png
> 
> This is IRQ:
> http://img7.imageshack.us/img7/259/irqstats.png
> 
> This is Interrupts:
> http://img851.imageshack.us/img851/1985/interrupts.png
> 
> This is eth0:
> http://img259.imageshack.us/img259/5730/ifeth0.png
> 
> 
> So, my questions about graphs are:
> 
> 1. Why my IRQ are so high? Especially "Rescheduling interrupts"? I read somewhere that it could be because of power management and it is expectable that server consumes more energy under the load, but I have to idea how to fix it in this case.
> 
> 2. Why "Interrupts" are so high?
> 
> Please, Erlangers, those of you who has experience with highly loaded erlang nodes tell me about your experience:
> 
> 1. Which values of IRQ and Interrupts are ok/normal? 
> 
> 2. Which values you see on your servers under high load?
> 
> 3. Which OS is the best for high computation/traffic to host Erlang/OTP application?
> 
> 4. Which settings you apply to you OS (general description or configs are welcome) to manage high load on Erlang?
> 
> 
> I hope that this information will be useful for all of us, because I believe everybody will face problems like I do one day when high load is applied.
> 
> Thank you all in advance!
> 
> Best regards,
> Max
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120130/0add1283/attachment.htm>