<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Now that I look more carefully, it looks like your system is
spending a whole lot of CPU time (~75%) in kernel space. Some system
call (or calls) is taking away way too many resources.<br>
<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 16:19, Guilherme Andrade
wrote:<br>
</div>
<blockquote cite="mid:5495A19E.2020405@gandrade.net" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
Hello gongtao,<br>
<br>
Besides 'message_queue_len', I would also look into 'reductions'
over time (also using process_info/2.)<br>
<br>
As for big message queues, if the CPU is not overloaded, in most
cases it's due to single processes becoming a bottleneck, whether
due to IO and/or heavy crunching; if they can be pooled or split
into multiple ones, the better. However, you say that CPU is
topping at 100%, so it's most likely that it's just too heavy.<br>
<br>
Long-running NIFs could also cause weird behaviour in the
scheduler.<br>
<br>
This dirty (multiple) one-liner should give you some insight into
reduction count:<br>
<br>
------------------<br>
rp([{-Reduc, Pid, case process_info(Pid, registered_name) of
{registered_name,Name} -> Name; _ -> '_' end} || <br>
{Reduc, Pid} <- <br>
lists:foldl( <br>
fun(Pid, L) when length(L) > 40 -> <br>
SL = lists:sublist(lists:keysort(1, L), 20), <br>
case process_info(Pid, reductions) of <br>
{reductions,Reduc} -> [{-Reduc, Pid} | SL];
<br>
undefined -> L <br>
end; <br>
(Pid, L) -> <br>
case process_info(Pid, reductions) of<br>
{reductions,Reduc} -> [{-Reduc, Pid} | L];
<br>
undefined -> L <br>
end <br>
end, [], erlang:processes())]).<br>
------------------<br>
<br>
<br>
Cheers,<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 12:03, gongtao wrote:<br>
</div>
<blockquote
cite="mid:4afb7459.10962.14a67946226.Coremail.goto310@163.com"
type="cite">
<div
style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">
<div>hello everyone:</div>
<div> I am a programmer from china.I have a problem with my
game server.There is 8193 processes in my erlang node.But
the cpu usage is nearly full,and the computer have 24 cpus.I
looked some erlang processes info,find the message_queue_len
of process is very big,more than 1000 sometimes.The disk io
and net io are normal,memory is normal.My erlang version is
R15B03.I don't know why.</div>
<div> My english is poor.There are same informations from
system.Hope you can understand.</div>
<div> Looking forward to your reply.</div>
<div> Thanks.</div>
<div><br>
</div>
<div>
<div>-bash-4.2$ sar -u 2 10</div>
<div>Linux 2.6.32-279.el6.x86_64 12/20/2014
_x86_64_ (24 CPU)</div>
<div><br>
</div>
<div>06:33:37 PM CPU %user %nice %system
%iowait %steal %idle</div>
<div>06:33:39 PM all 11.21 0.00 75.04
0.00 0.00 13.75</div>
<div>06:33:41 PM all 10.91 0.00 77.22
0.00 0.00 11.86</div>
<div>06:33:43 PM all 12.08 0.00 75.96
0.00 0.00 11.97</div>
<div>06:33:45 PM all 10.50 0.00 77.71
0.00 0.00 11.79</div>
<div>06:33:47 PM all 11.09 0.00 76.75
0.00 0.00 12.16</div>
<div>06:33:49 PM all 11.07 0.00 76.03
0.00 0.00 12.90</div>
<div>06:33:51 PM all 12.07 0.00 75.27
0.00 0.00 12.66</div>
<div>06:33:53 PM all 11.36 0.00 76.84
0.00 0.00 11.80</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>| CPU Utilisation
-------------------------------------------------------------------------------------------------------------------------------------
|</div>
<div>|---------------------------+-------------------------------------------------+
|</div>
<div>|CPU User% Sys% Wait% Idle|0 |25 |50
|75 100|
|</div>
<div>| 1 14.1 75.9 0.0
9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss >|
|</div>
<div>| 2 9.8 81.3 0.0
8.8|UUUUssssssssssssssssssssssssssssssssssssssss >|
|</div>
<div>| 3 11.6 76.9 0.0
11.6|UUUUUssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 4 7.6 82.3 0.0
10.1|UUUsssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 5 14.2 74.1 0.0
11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 6 8.6 81.3 0.0
10.1|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 7 11.6 75.9 0.0
12.6|UUUUUsssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 8 8.5 81.0 0.0
10.5|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 9 11.7 76.1 0.0
12.2|UUUUUssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 10 9.1 80.8 0.0
10.1|UUUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 11 10.2 76.5 0.0
13.3|UUUUUssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 12 6.6 81.6 0.0
11.7|UUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 13 12.1 75.3 0.0
12.6|UUUUUUsssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 14 9.0 80.6 0.0
10.4|UUUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 15 15.7 71.1 0.0
13.2|UUUUUUUsssssssssssssssssssssssssssssssssss
>|
|</div>
<div>| 16 9.2 80.1 0.0
10.7|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 17 13.6 74.2 0.0
12.1|UUUUUUsssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 18 7.2 82.1 0.0
10.8|UUUsssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 19 16.5 69.5 0.0
14.0|UUUUUUUUssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 20 7.3 81.2 0.0
11.5|UUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 21 17.8 69.0 0.0
13.2|UUUUUUUUssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 22 11.4 77.2 0.0
11.4|UUUUUssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 23 14.4 71.1 0.0
14.4|UUUUUUUsssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 24 5.8 81.7 0.0
12.6|UUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>|---------------------------+-------------------------------------------------+
|</div>
<div>|Avg 11.0 77.3 0.0
11.7|UUUUUssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>|---------------------------+-------------------------------------------------+
</div>
</div>
<div><br>
</div>
</div>
<br>
<br>
<span title="neteasefooter"><span id="netease_mail_footer"></span></span><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
</body>
</html>