[erlang-questions] High CPU usage problem
Guilherme Andrade
g@REDACTED
Sat Dec 20 17:53:04 CET 2014
Looking into kernel threads (both current CPU utilisation and consumed
CPU time) in something like ps / htop should help.
Cheers,
On 20-12-2014 16:49, Guilherme Andrade wrote:
> Now that I look more carefully, it looks like your system is spending
> a whole lot of CPU time (~75%) in kernel space. Some system call (or
> calls) is taking away way too many resources.
>
>
> On 20-12-2014 16:19, Guilherme Andrade wrote:
>> Hello gongtao,
>>
>> Besides 'message_queue_len', I would also look into 'reductions' over
>> time (also using process_info/2.)
>>
>> As for big message queues, if the CPU is not overloaded, in most
>> cases it's due to single processes becoming a bottleneck, whether due
>> to IO and/or heavy crunching; if they can be pooled or split into
>> multiple ones, the better. However, you say that CPU is topping at
>> 100%, so it's most likely that it's just too heavy.
>>
>> Long-running NIFs could also cause weird behaviour in the scheduler.
>>
>> This dirty (multiple) one-liner should give you some insight into
>> reduction count:
>>
>> ------------------
>> rp([{-Reduc, Pid, case process_info(Pid, registered_name) of
>> {registered_name,Name} -> Name; _ -> '_' end} ||
>> {Reduc, Pid} <-
>> lists:foldl(
>> fun(Pid, L) when length(L) > 40 ->
>> SL = lists:sublist(lists:keysort(1, L), 20),
>> case process_info(Pid, reductions) of
>> {reductions,Reduc} -> [{-Reduc, Pid} | SL];
>> undefined -> L
>> end;
>> (Pid, L) ->
>> case process_info(Pid, reductions) of
>> {reductions,Reduc} -> [{-Reduc, Pid} | L];
>> undefined -> L
>> end
>> end, [], erlang:processes())]).
>> ------------------
>>
>>
>> Cheers,
>>
>> On 20-12-2014 12:03, gongtao wrote:
>>> hello everyone:
>>> I am a programmer from china.I have a problem with my game
>>> server.There is 8193 processes in my erlang node.But the cpu usage
>>> is nearly full,and the computer have 24 cpus.I looked some erlang
>>> processes info,find the message_queue_len of process is very
>>> big,more than 1000 sometimes.The disk io and net io are
>>> normal,memory is normal.My erlang version is R15B03.I don't know why.
>>> My english is poor.There are same informations from system.Hope
>>> you can understand.
>>> Looking forward to your reply.
>>> Thanks.
>>>
>>> -bash-4.2$ sar -u 2 10
>>> Linux 2.6.32-279.el6.x86_64 12/20/2014 _x86_64_
>>> (24 CPU)
>>>
>>> 06:33:37 PM CPU %user %nice %system %iowait
>>> %steal %idle
>>> 06:33:39 PM all 11.21 0.00 75.04 0.00
>>> 0.00 13.75
>>> 06:33:41 PM all 10.91 0.00 77.22 0.00
>>> 0.00 11.86
>>> 06:33:43 PM all 12.08 0.00 75.96 0.00
>>> 0.00 11.97
>>> 06:33:45 PM all 10.50 0.00 77.71 0.00
>>> 0.00 11.79
>>> 06:33:47 PM all 11.09 0.00 76.75 0.00
>>> 0.00 12.16
>>> 06:33:49 PM all 11.07 0.00 76.03 0.00
>>> 0.00 12.90
>>> 06:33:51 PM all 12.07 0.00 75.27 0.00
>>> 0.00 12.66
>>> 06:33:53 PM all 11.36 0.00 76.84 0.00
>>> 0.00 11.80
>>>
>>>
>>> | CPU Utilisation
>>> -------------------------------------------------------------------------------------------------------------------------------------
>>> |
>>> |---------------------------+-------------------------------------------------+
>>>
>>> |
>>> |CPU User% Sys% Wait% Idle|0 |25 |50 |75
>>> 100|
>>> |
>>> | 1 14.1 75.9 0.0
>>> 9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss >|
>>> |
>>> | 2 9.8 81.3 0.0
>>> 8.8|UUUUssssssssssssssssssssssssssssssssssssssss >|
>>> |
>>> | 3 11.6 76.9 0.0
>>> 11.6|UUUUUssssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 4 7.6 82.3 0.0
>>> 10.1|UUUsssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 5 14.2 74.1 0.0
>>> 11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 6 8.6 81.3 0.0
>>> 10.1|UUUUssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 7 11.6 75.9 0.0
>>> 12.6|UUUUUsssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 8 8.5 81.0 0.0
>>> 10.5|UUUUssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 9 11.7 76.1 0.0
>>> 12.2|UUUUUssssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 10 9.1 80.8 0.0
>>> 10.1|UUUUssssssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 11 10.2 76.5 0.0
>>> 13.3|UUUUUssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 12 6.6 81.6 0.0
>>> 11.7|UUUssssssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 13 12.1 75.3 0.0
>>> 12.6|UUUUUUsssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 14 9.0 80.6 0.0
>>> 10.4|UUUUssssssssssssssssssssssssssssssssssssssss >
>>> |
>>> | 15 15.7 71.1 0.0
>>> 13.2|UUUUUUUsssssssssssssssssssssssssssssssssss >|
>>> |
>>> | 16 9.2 80.1 0.0
>>> 10.7|UUUUssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 17 13.6 74.2 0.0
>>> 12.1|UUUUUUsssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 18 7.2 82.1 0.0
>>> 10.8|UUUsssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 19 16.5 69.5 0.0
>>> 14.0|UUUUUUUUssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 20 7.3 81.2 0.0
>>> 11.5|UUUssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 21 17.8 69.0 0.0
>>> 13.2|UUUUUUUUssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 22 11.4 77.2 0.0
>>> 11.4|UUUUUssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 23 14.4 71.1 0.0
>>> 14.4|UUUUUUUsssssssssssssssssssssssssssssssssss > |
>>> |
>>> | 24 5.8 81.7 0.0
>>> 12.6|UUssssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> |---------------------------+-------------------------------------------------+
>>>
>>> |
>>> |Avg 11.0 77.3 0.0
>>> 11.7|UUUUUssssssssssssssssssssssssssssssssssssss > |
>>> |
>>> |---------------------------+-------------------------------------------------+
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>> --
>> Guilherme
>>
>> https://www.gandrade.net/
>> PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>
> --
> Guilherme
>
> https://www.gandrade.net/
> PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
--
Guilherme
https://www.gandrade.net/
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141220/f7903573/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: OpenPGP digital signature
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141220/f7903573/attachment.bin>
More information about the erlang-questions
mailing list