[erlang-questions] High CPU usage problem
gongtao
goto310@REDACTED
Tue Dec 23 11:03:29 CET 2014
Thanks for you reply.
I got more informations from system,there is the CPU analyze result,looks like the most cost is sched_yield.What calls in erlang can trigger a sched_yield?
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
83.95 51.691294 18 2917715 sched_yield
13.23 8.147366 60 136096 53085 futex
0.92 0.568038 18 31832 4954 recvfrom
0.91 0.559971 16 35466 51 writev
0.29 0.177944 14 12999 5 stat
0.28 0.172864 16 10555 196 epoll_ctl
0.16 0.095549 95549 1 vfork
0.08 0.048692 73 666 munmap
0.07 0.046025 1 77373 74974 close
0.07 0.045406 16 2815 epoll_wait
0.01 0.008748 35 253 mmap
0.01 0.004698 36 132 getdents
0.01 0.003173 20 156 getsockopt
0.00 0.001910 18 104 setsockopt
0.00 0.001204 16 74 getpeername
0.00 0.000997 13 75 open
0.00 0.000336 8 42 fcntl
0.00 0.000136 7 20 1 accept
0.00 0.000000 0 1 read
0.00 0.000000 0 5 write
0.00 0.000000 0 2 fstat
0.00 0.000000 0 3 mprotect
0.00 0.000000 0 1 brk
0.00 0.000000 0 13 1 access
0.00 0.000000 0 2 pipe
0.00 0.000000 0 2 dup2
0.00 0.000000 0 1 execve
0.00 0.000000 0 1 getcwd
0.00 0.000000 0 1 times
0.00 0.000000 0 1 arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00 61.574351 3226407 133267 total
In erlang node there is a process called 'erl_prim_loader',this process has the biggest reduction,is it normall?
[{registered_name,erl_prim_loader},
{current_function,{erl_prim_loader,loop,3}},
{initial_call,{erlang,apply,2}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[#Port<0.1>,<0.0.0>]},
{dictionary,[]},
{trap_exit,true},
{error_handler,error_handler},
{priority,normal},
{group_leader,<0.0.0>},
{total_heap_size,57314},
{heap_size,10946},
{stack_size,6},
{reductions,7774178639},
{garbage_collection,[{min_bin_vheap_size,10946},
{min_heap_size,10946},
{fullsweep_after,65535},
{minor_gcs,12}]},
{suspending,[]}]
At 2014-12-21 00:53:04, "Guilherme Andrade" <g@REDACTED> wrote:
Looking into kernel threads (both current CPU utilisation and consumed CPU time) in something like ps / htop should help.
Cheers,
On 20-12-2014 16:49, Guilherme Andrade wrote:
Now that I look more carefully, it looks like your system is spending a whole lot of CPU time (~75%) in kernel space. Some system call (or calls) is taking away way too many resources.
On 20-12-2014 16:19, Guilherme Andrade wrote:
Hello gongtao,
Besides 'message_queue_len', I would also look into 'reductions' over time (also using process_info/2.)
As for big message queues, if the CPU is not overloaded, in most cases it's due to single processes becoming a bottleneck, whether due to IO and/or heavy crunching; if they can be pooled or split into multiple ones, the better. However, you say that CPU is topping at 100%, so it's most likely that it's just too heavy.
Long-running NIFs could also cause weird behaviour in the scheduler.
This dirty (multiple) one-liner should give you some insight into reduction count:
------------------
rp([{-Reduc, Pid, case process_info(Pid, registered_name) of {registered_name,Name} -> Name; _ -> '_' end} ||
{Reduc, Pid} <-
lists:foldl(
fun(Pid, L) when length(L) > 40 ->
SL = lists:sublist(lists:keysort(1, L), 20),
case process_info(Pid, reductions) of
{reductions,Reduc} -> [{-Reduc, Pid} | SL];
undefined -> L
end;
(Pid, L) ->
case process_info(Pid, reductions) of
{reductions,Reduc} -> [{-Reduc, Pid} | L];
undefined -> L
end
end, [], erlang:processes())]).
------------------
Cheers,
On 20-12-2014 12:03, gongtao wrote:
hello everyone:
I am a programmer from china.I have a problem with my game server.There is 8193 processes in my erlang node.But the cpu usage is nearly full,and the computer have 24 cpus.I looked some erlang processes info,find the message_queue_len of process is very big,more than 1000 sometimes.The disk io and net io are normal,memory is normal.My erlang version is R15B03.I don't know why.
My english is poor.There are same informations from system.Hope you can understand.
Looking forward to your reply.
Thanks.
-bash-4.2$ sar -u 2 10
Linux 2.6.32-279.el6.x86_64 12/20/2014 _x86_64_ (24 CPU)
06:33:37 PM CPU %user %nice %system %iowait %steal %idle
06:33:39 PM all 11.21 0.00 75.04 0.00 0.00 13.75
06:33:41 PM all 10.91 0.00 77.22 0.00 0.00 11.86
06:33:43 PM all 12.08 0.00 75.96 0.00 0.00 11.97
06:33:45 PM all 10.50 0.00 77.71 0.00 0.00 11.79
06:33:47 PM all 11.09 0.00 76.75 0.00 0.00 12.16
06:33:49 PM all 11.07 0.00 76.03 0.00 0.00 12.90
06:33:51 PM all 12.07 0.00 75.27 0.00 0.00 12.66
06:33:53 PM all 11.36 0.00 76.84 0.00 0.00 11.80
| CPU Utilisation ------------------------------------------------------------------------------------------------------------------------------------- |
|---------------------------+-------------------------------------------------+ |
|CPU User% Sys% Wait% Idle|0 |25 |50 |75 100| |
| 1 14.1 75.9 0.0 9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss >| |
| 2 9.8 81.3 0.0 8.8|UUUUssssssssssssssssssssssssssssssssssssssss >| |
| 3 11.6 76.9 0.0 11.6|UUUUUssssssssssssssssssssssssssssssssssssss > |
| 4 7.6 82.3 0.0 10.1|UUUsssssssssssssssssssssssssssssssssssssssss > | |
| 5 14.2 74.1 0.0 11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss > | |
| 6 8.6 81.3 0.0 10.1|UUUUssssssssssssssssssssssssssssssssssssssss > | |
| 7 11.6 75.9 0.0 12.6|UUUUUsssssssssssssssssssssssssssssssssssss > |
| 8 8.5 81.0 0.0 10.5|UUUUssssssssssssssssssssssssssssssssssssssss > | |
| 9 11.7 76.1 0.0 12.2|UUUUUssssssssssssssssssssssssssssssssssssss > |
| 10 9.1 80.8 0.0 10.1|UUUUssssssssssssssssssssssssssssssssssssssss > |
| 11 10.2 76.5 0.0 13.3|UUUUUssssssssssssssssssssssssssssssssssssss > | |
| 12 6.6 81.6 0.0 11.7|UUUssssssssssssssssssssssssssssssssssssssss > |
| 13 12.1 75.3 0.0 12.6|UUUUUUsssssssssssssssssssssssssssssssssssss > |
| 14 9.0 80.6 0.0 10.4|UUUUssssssssssssssssssssssssssssssssssssssss > |
| 15 15.7 71.1 0.0 13.2|UUUUUUUsssssssssssssssssssssssssssssssssss >| |
| 16 9.2 80.1 0.0 10.7|UUUUssssssssssssssssssssssssssssssssssssssss > | |
| 17 13.6 74.2 0.0 12.1|UUUUUUsssssssssssssssssssssssssssssssssssss > | |
| 18 7.2 82.1 0.0 10.8|UUUsssssssssssssssssssssssssssssssssssssssss > | |
| 19 16.5 69.5 0.0 14.0|UUUUUUUUssssssssssssssssssssssssssssssssss > | |
| 20 7.3 81.2 0.0 11.5|UUUssssssssssssssssssssssssssssssssssssssss > | |
| 21 17.8 69.0 0.0 13.2|UUUUUUUUssssssssssssssssssssssssssssssssss > | |
| 22 11.4 77.2 0.0 11.4|UUUUUssssssssssssssssssssssssssssssssssssss > | |
| 23 14.4 71.1 0.0 14.4|UUUUUUUsssssssssssssssssssssssssssssssssss > | |
| 24 5.8 81.7 0.0 12.6|UUssssssssssssssssssssssssssssssssssssssss > | |
|---------------------------+-------------------------------------------------+ |
|Avg 11.0 77.3 0.0 11.7|UUUUUssssssssssssssssssssssssssssssssssssss > | |
|---------------------------+-------------------------------------------------+
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
--
Guilherme
https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
--
Guilherme
https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED://erlang.org/mailman/listinfo/erlang-questions
--
Guilherme
https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141223/b3219879/attachment.htm>
More information about the erlang-questions
mailing list