[erlang-questions] High CPU usage problem

gongtao <>
Tue Dec 23 11:03:29 CET 2014


Thanks for you reply.


I got more informations from system,there is the CPU analyze result,looks like the most cost is sched_yield.What calls in erlang can trigger a sched_yield?
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 83.95   51.691294          18   2917715           sched_yield
 13.23    8.147366          60    136096     53085 futex
  0.92    0.568038          18     31832      4954 recvfrom
  0.91    0.559971          16     35466        51 writev
  0.29    0.177944          14     12999         5 stat
  0.28    0.172864          16     10555       196 epoll_ctl
  0.16    0.095549       95549         1           vfork
  0.08    0.048692          73       666           munmap
  0.07    0.046025           1     77373     74974 close
  0.07    0.045406          16      2815           epoll_wait
  0.01    0.008748          35       253           mmap
  0.01    0.004698          36       132           getdents
  0.01    0.003173          20       156           getsockopt
  0.00    0.001910          18       104           setsockopt
  0.00    0.001204          16        74           getpeername
  0.00    0.000997          13        75           open
  0.00    0.000336           8        42           fcntl
  0.00    0.000136           7        20         1 accept
  0.00    0.000000           0         1           read
  0.00    0.000000           0         5           write
  0.00    0.000000           0         2           fstat
  0.00    0.000000           0         3           mprotect
  0.00    0.000000           0         1           brk
  0.00    0.000000           0        13         1 access
  0.00    0.000000           0         2           pipe
  0.00    0.000000           0         2           dup2
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           getcwd
  0.00    0.000000           0         1           times
  0.00    0.000000           0         1           arch_prctl
------ ----------- ----------- --------- --------- ----------------
100.00   61.574351               3226407    133267 total

In erlang node there is a process called 'erl_prim_loader',this process has the biggest reduction,is it normall?

[{registered_name,erl_prim_loader},
 {current_function,{erl_prim_loader,loop,3}},
 {initial_call,{erlang,apply,2}},
 {status,waiting},
 {message_queue_len,0},
 {messages,[]},
 {links,[#Port<0.1>,<0.0.0>]},
 {dictionary,[]},
 {trap_exit,true},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.0.0>},
 {total_heap_size,57314},
 {heap_size,10946},
 {stack_size,6},
 {reductions,7774178639},
 {garbage_collection,[{min_bin_vheap_size,10946},
                      {min_heap_size,10946},
                      {fullsweep_after,65535},
                      {minor_gcs,12}]},
 {suspending,[]}]



At 2014-12-21 00:53:04, "Guilherme Andrade" <> wrote:
Looking into kernel threads (both current CPU utilisation and consumed CPU time) in something like ps / htop should help.

Cheers,



On 20-12-2014 16:49, Guilherme Andrade wrote:

Now that I look more carefully, it looks like your system is spending a whole lot of CPU time (~75%) in kernel space. Some system call (or calls) is taking away way too many resources.



On 20-12-2014 16:19, Guilherme Andrade wrote:

Hello gongtao,

Besides 'message_queue_len', I would also look into 'reductions' over time (also using process_info/2.)

As for big message queues, if the CPU is not overloaded, in most cases it's due to single processes becoming a bottleneck, whether due to IO and/or heavy crunching; if they can be pooled or split into multiple ones, the better. However, you say that CPU is topping at 100%, so it's most likely that it's just too heavy.

Long-running NIFs could also cause weird behaviour in the scheduler.

This dirty (multiple) one-liner should give you some insight into reduction count:

------------------
rp([{-Reduc, Pid, case process_info(Pid, registered_name) of {registered_name,Name} -> Name; _ -> '_' end} ||
    {Reduc, Pid} <-
    lists:foldl(
        fun(Pid, L) when length(L) > 40 ->
                SL = lists:sublist(lists:keysort(1, L), 20),
                case process_info(Pid, reductions) of
                    {reductions,Reduc} -> [{-Reduc, Pid} | SL];
                    undefined -> L
                end;
            (Pid, L) ->
                case process_info(Pid, reductions) of
                    {reductions,Reduc} -> [{-Reduc, Pid} | L];
                    undefined -> L
                end
        end, [], erlang:processes())]).
------------------


Cheers,


On 20-12-2014 12:03, gongtao wrote:

hello everyone:
    I am a programmer from china.I have a problem with my game server.There is 8193 processes in my erlang node.But the cpu usage is nearly full,and the computer have 24 cpus.I looked some erlang processes info,find the message_queue_len of process is very big,more than 1000 sometimes.The disk io and net io are normal,memory is normal.My erlang version is R15B03.I don't know why.
    My english is poor.There are same informations from system.Hope you can understand.
    Looking forward to your reply.
    Thanks.


-bash-4.2$ sar -u 2 10
Linux 2.6.32-279.el6.x86_64          12/20/2014      _x86_64_        (24 CPU)


06:33:37 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
06:33:39 PM     all     11.21      0.00     75.04      0.00      0.00     13.75
06:33:41 PM     all     10.91      0.00     77.22      0.00      0.00     11.86
06:33:43 PM     all     12.08      0.00     75.96      0.00      0.00     11.97
06:33:45 PM     all     10.50      0.00     77.71      0.00      0.00     11.79
06:33:47 PM     all     11.09      0.00     76.75      0.00      0.00     12.16
06:33:49 PM     all     11.07      0.00     76.03      0.00      0.00     12.90
06:33:51 PM     all     12.07      0.00     75.27      0.00      0.00     12.66
06:33:53 PM     all     11.36      0.00     76.84      0.00      0.00     11.80




| CPU Utilisation -------------------------------------------------------------------------------------------------------------------------------------     |
|---------------------------+-------------------------------------------------+                                                                             |
|CPU  User%  Sys% Wait% Idle|0          |25         |50          |75       100|                                                                             |
|  1  14.1  75.9   0.0   9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss    >|                                                                             |
|  2   9.8  81.3   0.0   8.8|UUUUssssssssssssssssssssssssssssssssssssssss    >|                                                                             |
|  3  11.6  76.9   0.0  11.6|UUUUUssssssssssssssssssssssssssssssssssssss      >                                                                             |
|  4   7.6  82.3   0.0  10.1|UUUsssssssssssssssssssssssssssssssssssssssss >   |                                                                             |
|  5  14.2  74.1   0.0  11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss  >  |                                                                             |
|  6   8.6  81.3   0.0  10.1|UUUUssssssssssssssssssssssssssssssssssssssss  >  |                                                                             |
|  7  11.6  75.9   0.0  12.6|UUUUUsssssssssssssssssssssssssssssssssssss       >                                                                             |
|  8   8.5  81.0   0.0  10.5|UUUUssssssssssssssssssssssssssssssssssssssss >   |                                                                             |
|  9  11.7  76.1   0.0  12.2|UUUUUssssssssssssssssssssssssssssssssssssss      >                                                                             |
| 10   9.1  80.8   0.0  10.1|UUUUssssssssssssssssssssssssssssssssssssssss     >                                                                             |
| 11  10.2  76.5   0.0  13.3|UUUUUssssssssssssssssssssssssssssssssssssss    > |                                                                             |
| 12   6.6  81.6   0.0  11.7|UUUssssssssssssssssssssssssssssssssssssssss      >                                                                             |
| 13  12.1  75.3   0.0  12.6|UUUUUUsssssssssssssssssssssssssssssssssssss      >                                                                             |
| 14   9.0  80.6   0.0  10.4|UUUUssssssssssssssssssssssssssssssssssssssss     >                                                                             |
| 15  15.7  71.1   0.0  13.2|UUUUUUUsssssssssssssssssssssssssssssssssss      >|                                                                             |
| 16   9.2  80.1   0.0  10.7|UUUUssssssssssssssssssssssssssssssssssssssss  >  |                                                                             |
| 17  13.6  74.2   0.0  12.1|UUUUUUsssssssssssssssssssssssssssssssssssss   >  |                                                                             |
| 18   7.2  82.1   0.0  10.8|UUUsssssssssssssssssssssssssssssssssssssssss >   |                                                                             |
| 19  16.5  69.5   0.0  14.0|UUUUUUUUssssssssssssssssssssssssssssssssss   >   |                                                                             |
| 20   7.3  81.2   0.0  11.5|UUUssssssssssssssssssssssssssssssssssssssss    > |                                                                             |
| 21  17.8  69.0   0.0  13.2|UUUUUUUUssssssssssssssssssssssssssssssssss     > |                                                                             |
| 22  11.4  77.2   0.0  11.4|UUUUUssssssssssssssssssssssssssssssssssssss >    |                                                                             |
| 23  14.4  71.1   0.0  14.4|UUUUUUUsssssssssssssssssssssssssssssssssss   >   |                                                                             |
| 24   5.8  81.7   0.0  12.6|UUssssssssssssssssssssssssssssssssssssssss >     |                                                                             |
|---------------------------+-------------------------------------------------+                                                                             |
|Avg  11.0  77.3   0.0  11.7|UUUUUssssssssssssssssssssssssssssssssssssss >    |                                                                             |
|---------------------------+-------------------------------------------------+  







_______________________________________________
erlang-questions mailing list
://erlang.org/mailman/listinfo/erlang-questions


-- 
Guilherme

https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191




_______________________________________________
erlang-questions mailing list
://erlang.org/mailman/listinfo/erlang-questions


-- 
Guilherme

https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191




_______________________________________________
erlang-questions mailing list
://erlang.org/mailman/listinfo/erlang-questions


-- 
Guilherme

https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141223/b3219879/attachment-0001.html>


More information about the erlang-questions mailing list