[erlang-questions] High CPU usage problem

Guilherme Andrade g@REDACTED
Sat Dec 20 17:53:04 CET 2014


Looking into kernel threads (both current CPU utilisation and consumed
CPU time) in something like ps / htop should help.

Cheers,


On 20-12-2014 16:49, Guilherme Andrade wrote:
> Now that I look more carefully, it looks like your system is spending
> a whole lot of CPU time (~75%) in kernel space. Some system call (or
> calls) is taking away way too many resources.
>
>
> On 20-12-2014 16:19, Guilherme Andrade wrote:
>> Hello gongtao,
>>
>> Besides 'message_queue_len', I would also look into 'reductions' over
>> time (also using process_info/2.)
>>
>> As for big message queues, if the CPU is not overloaded, in most
>> cases it's due to single processes becoming a bottleneck, whether due
>> to IO and/or heavy crunching; if they can be pooled or split into
>> multiple ones, the better. However, you say that CPU is topping at
>> 100%, so it's most likely that it's just too heavy.
>>
>> Long-running NIFs could also cause weird behaviour in the scheduler.
>>
>> This dirty (multiple) one-liner should give you some insight into
>> reduction count:
>>
>> ------------------
>> rp([{-Reduc, Pid, case process_info(Pid, registered_name) of
>> {registered_name,Name} -> Name; _ -> '_' end} ||
>>     {Reduc, Pid} <-
>>     lists:foldl(
>>         fun(Pid, L) when length(L) > 40 ->
>>                 SL = lists:sublist(lists:keysort(1, L), 20),
>>                 case process_info(Pid, reductions) of
>>                     {reductions,Reduc} -> [{-Reduc, Pid} | SL];
>>                     undefined -> L
>>                 end;
>>             (Pid, L) ->
>>                 case process_info(Pid, reductions) of
>>                     {reductions,Reduc} -> [{-Reduc, Pid} | L];
>>                     undefined -> L
>>                 end
>>         end, [], erlang:processes())]).
>> ------------------
>>
>>
>> Cheers,
>>
>> On 20-12-2014 12:03, gongtao wrote:
>>> hello everyone:
>>>     I am a programmer from china.I have a problem with my game
>>> server.There is 8193 processes in my erlang node.But the cpu usage
>>> is nearly full,and the computer have 24 cpus.I looked some erlang
>>> processes info,find the message_queue_len of process is very
>>> big,more than 1000 sometimes.The disk io and net io are
>>> normal,memory is normal.My erlang version is R15B03.I don't know why.
>>>     My english is poor.There are same informations from system.Hope
>>> you can understand.
>>>     Looking forward to your reply.
>>>     Thanks.
>>>
>>> -bash-4.2$ sar -u 2 10
>>> Linux 2.6.32-279.el6.x86_64          12/20/2014      _x86_64_      
>>>  (24 CPU)
>>>
>>> 06:33:37 PM     CPU     %user     %nice   %system   %iowait  
>>>  %steal     %idle
>>> 06:33:39 PM     all     11.21      0.00     75.04      0.00    
>>>  0.00     13.75
>>> 06:33:41 PM     all     10.91      0.00     77.22      0.00    
>>>  0.00     11.86
>>> 06:33:43 PM     all     12.08      0.00     75.96      0.00    
>>>  0.00     11.97
>>> 06:33:45 PM     all     10.50      0.00     77.71      0.00    
>>>  0.00     11.79
>>> 06:33:47 PM     all     11.09      0.00     76.75      0.00    
>>>  0.00     12.16
>>> 06:33:49 PM     all     11.07      0.00     76.03      0.00    
>>>  0.00     12.90
>>> 06:33:51 PM     all     12.07      0.00     75.27      0.00    
>>>  0.00     12.66
>>> 06:33:53 PM     all     11.36      0.00     76.84      0.00    
>>>  0.00     11.80
>>>
>>>
>>> | CPU Utilisation
>>> -------------------------------------------------------------------------------------------------------------------------------------
>>>     |
>>> |---------------------------+-------------------------------------------------+
>>>                                                                    
>>>         |
>>> |CPU  User%  Sys% Wait% Idle|0          |25         |50          |75
>>>       100|                                                          
>>>                   |
>>> |  1  14.1  75.9   0.0  
>>> 9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss    >|              
>>>                                                               |
>>> |  2   9.8  81.3   0.0  
>>> 8.8|UUUUssssssssssssssssssssssssssssssssssssssss    >|              
>>>                                                               |
>>> |  3  11.6  76.9   0.0
>>>  11.6|UUUUUssssssssssssssssssssssssssssssssssssss      >            
>>>                                                                 |
>>> |  4   7.6  82.3   0.0
>>>  10.1|UUUsssssssssssssssssssssssssssssssssssssssss >   |            
>>>                                                                 |
>>> |  5  14.2  74.1   0.0
>>>  11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss  >  |            
>>>                                                                 |
>>> |  6   8.6  81.3   0.0
>>>  10.1|UUUUssssssssssssssssssssssssssssssssssssssss  >  |            
>>>                                                                 |
>>> |  7  11.6  75.9   0.0
>>>  12.6|UUUUUsssssssssssssssssssssssssssssssssssss       >            
>>>                                                                 |
>>> |  8   8.5  81.0   0.0
>>>  10.5|UUUUssssssssssssssssssssssssssssssssssssssss >   |            
>>>                                                                 |
>>> |  9  11.7  76.1   0.0
>>>  12.2|UUUUUssssssssssssssssssssssssssssssssssssss      >            
>>>                                                                 |
>>> | 10   9.1  80.8   0.0
>>>  10.1|UUUUssssssssssssssssssssssssssssssssssssssss     >            
>>>                                                                 |
>>> | 11  10.2  76.5   0.0
>>>  13.3|UUUUUssssssssssssssssssssssssssssssssssssss    > |            
>>>                                                                 |
>>> | 12   6.6  81.6   0.0
>>>  11.7|UUUssssssssssssssssssssssssssssssssssssssss      >            
>>>                                                                 |
>>> | 13  12.1  75.3   0.0
>>>  12.6|UUUUUUsssssssssssssssssssssssssssssssssssss      >            
>>>                                                                 |
>>> | 14   9.0  80.6   0.0
>>>  10.4|UUUUssssssssssssssssssssssssssssssssssssssss     >            
>>>                                                                 |
>>> | 15  15.7  71.1   0.0
>>>  13.2|UUUUUUUsssssssssssssssssssssssssssssssssss      >|            
>>>                                                                 |
>>> | 16   9.2  80.1   0.0
>>>  10.7|UUUUssssssssssssssssssssssssssssssssssssssss  >  |            
>>>                                                                 |
>>> | 17  13.6  74.2   0.0
>>>  12.1|UUUUUUsssssssssssssssssssssssssssssssssssss   >  |            
>>>                                                                 |
>>> | 18   7.2  82.1   0.0
>>>  10.8|UUUsssssssssssssssssssssssssssssssssssssssss >   |            
>>>                                                                 |
>>> | 19  16.5  69.5   0.0
>>>  14.0|UUUUUUUUssssssssssssssssssssssssssssssssss   >   |            
>>>                                                                 |
>>> | 20   7.3  81.2   0.0
>>>  11.5|UUUssssssssssssssssssssssssssssssssssssssss    > |            
>>>                                                                 |
>>> | 21  17.8  69.0   0.0
>>>  13.2|UUUUUUUUssssssssssssssssssssssssssssssssss     > |            
>>>                                                                 |
>>> | 22  11.4  77.2   0.0
>>>  11.4|UUUUUssssssssssssssssssssssssssssssssssssss >    |            
>>>                                                                 |
>>> | 23  14.4  71.1   0.0
>>>  14.4|UUUUUUUsssssssssssssssssssssssssssssssssss   >   |            
>>>                                                                 |
>>> | 24   5.8  81.7   0.0
>>>  12.6|UUssssssssssssssssssssssssssssssssssssssss >     |            
>>>                                                                 |
>>> |---------------------------+-------------------------------------------------+
>>>                                                                    
>>>         |
>>> |Avg  11.0  77.3   0.0
>>>  11.7|UUUUUssssssssssssssssssssssssssssssssssssss >    |            
>>>                                                                 |
>>> |---------------------------+-------------------------------------------------+
>>>  
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>> -- 
>> Guilherme
>>
>> https://www.gandrade.net/
>> PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
> -- 
> Guilherme
>
> https://www.gandrade.net/
> PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-- 
Guilherme

https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A  D67A 9330 79B1 35CB 8191

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141220/f7903573/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: OpenPGP digital signature
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141220/f7903573/attachment.bin>


More information about the erlang-questions mailing list