<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Now that I look more carefully, it looks like your system is
spending a whole lot of CPU time (~75%) in kernel space. Some system
call (or calls) is taking away way too many resources.<br>
<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 16:19, Guilherme Andrade
wrote:<br>
</div>
<blockquote cite="mid:5495A19E.2020405@gandrade.net" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
Hello gongtao,<br>
<br>
Besides 'message_queue_len', I would also look into 'reductions'
over time (also using process_info/2.)<br>
<br>
As for big message queues, if the CPU is not overloaded, in most
cases it's due to single processes becoming a bottleneck, whether
due to IO and/or heavy crunching; if they can be pooled or split
into multiple ones, the better. However, you say that CPU is
topping at 100%, so it's most likely that it's just too heavy.<br>
<br>
Long-running NIFs could also cause weird behaviour in the
scheduler.<br>
<br>
This dirty (multiple) one-liner should give you some insight into
reduction count:<br>
<br>
------------------<br>
rp([{-Reduc, Pid, case process_info(Pid, registered_name) of
{registered_name,Name} -> Name; _ -> '_' end} || <br>
   {Reduc, Pid} <- <br>
   lists:foldl( <br>
      fun(Pid, L) when length(L) > 40 -> <br>
            SL = lists:sublist(lists:keysort(1, L), 20), <br>
            case process_info(Pid, reductions) of <br>
               {reductions,Reduc} -> [{-Reduc, Pid} | SL];
<br>
               undefined -> L <br>
            end; <br>
         (Pid, L) -> <br>
            case process_info(Pid, reductions) of<br>
               {reductions,Reduc} -> [{-Reduc, Pid} | L];
<br>
               undefined -> L <br>
            end <br>
      end, [], erlang:processes())]).<br>
------------------<br>
<br>
<br>
Cheers,<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 12:03, gongtao wrote:<br>
</div>
<blockquote
cite="mid:4afb7459.10962.14a67946226.Coremail.goto310@163.com"
type="cite">
<div
style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">
<div>hello everyone:</div>
<div>  I am a programmer from china.I have a problem with my
game server.There is 8193 processes in my erlang node.But
the cpu usage is nearly full,and the computer have 24 cpus.I
looked some erlang processes info,find the message_queue_len
of process is very big,more than 1000 sometimes.The disk io
and net io are normal,memory is normal.My erlang version is
R15B03.I don't know why.</div>
<div>  My english is poor.There are same informations from
system.Hope you can understand.</div>
<div>Â Â Â Looking forward to your reply.</div>
<div>Â Â Thanks.</div>
<div><br>
</div>
<div>
<div>-bash-4.2$ sar -u 2 10</div>
<div>Linux 2.6.32-279.el6.x86_64 Â Â Â Â Â 12/20/2014 Â Â
 _x86_64_     (24 CPU)</div>
<div><br>
</div>
<div>06:33:37 PM   CPU   %user   %nice  %system Â
%iowait   %steal   %idle</div>
<div>06:33:39 PM   all   11.21    0.00   75.04  Â
 0.00    0.00   13.75</div>
<div>06:33:41 PM   all   10.91    0.00   77.22  Â
 0.00    0.00   11.86</div>
<div>06:33:43 PM   all   12.08    0.00   75.96  Â
 0.00    0.00   11.97</div>
<div>06:33:45 PM   all   10.50    0.00   77.71  Â
 0.00    0.00   11.79</div>
<div>06:33:47 PM   all   11.09    0.00   76.75  Â
 0.00    0.00   12.16</div>
<div>06:33:49 PM   all   11.07    0.00   76.03  Â
 0.00    0.00   12.90</div>
<div>06:33:51 PM   all   12.07    0.00   75.27  Â
 0.00    0.00   12.66</div>
<div>06:33:53 PM   all   11.36    0.00   76.84  Â
 0.00    0.00   11.80</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>| CPU Utilisation
-------------------------------------------------------------------------------------------------------------------------------------
  |</div>
<div>|---------------------------+-------------------------------------------------+
                            Â
         |</div>
<div>|CPU Â User% Â Sys% Wait% Idle|0 Â Â Â Â Â |25 Â Â Â Â |50
     |75    100|                 Â
                     |</div>
<div>| Â 1 Â 14.1 Â 75.9 Â 0.0 Â
9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss   >|
                            Â
         |</div>
<div>| Â 2 Â 9.8 Â 81.3 Â 0.0 Â
8.8|UUUUssssssssssssssssssssssssssssssssssssssss   >|
                            Â
         |</div>
<div>| Â 3 Â 11.6 Â 76.9 Â 0.0
 11.6|UUUUUssssssssssssssssssssssssssssssssssssss  Â
 >                          Â
            |</div>
<div>| Â 4 Â 7.6 Â 82.3 Â 0.0
 10.1|UUUsssssssssssssssssssssssssssssssssssssssss > Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| Â 5 Â 14.2 Â 74.1 Â 0.0
 11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss  >
 |                            Â
          |</div>
<div>| Â 6 Â 8.6 Â 81.3 Â 0.0
 10.1|UUUUssssssssssssssssssssssssssssssssssssssss  >
 |                            Â
          |</div>
<div>| Â 7 Â 11.6 Â 75.9 Â 0.0
 12.6|UUUUUsssssssssssssssssssssssssssssssssssss   Â
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
           |</div>
<div>| Â 8 Â 8.5 Â 81.0 Â 0.0
 10.5|UUUUssssssssssssssssssssssssssssssssssssssss > Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| Â 9 Â 11.7 Â 76.1 Â 0.0
 12.2|UUUUUssssssssssssssssssssssssssssssssssssss  Â
 >                          Â
            |</div>
<div>| 10 Â 9.1 Â 80.8 Â 0.0
 10.1|UUUUssssssssssssssssssssssssssssssssssssssss  Â
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
           |</div>
<div>| 11 Â 10.2 Â 76.5 Â 0.0
 13.3|UUUUUssssssssssssssssssssssssssssssssssssss   >
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 12 Â 6.6 Â 81.6 Â 0.0
 11.7|UUUssssssssssssssssssssssssssssssssssssssss  Â
 >                          Â
            |</div>
<div>| 13 Â 12.1 Â 75.3 Â 0.0
 12.6|UUUUUUsssssssssssssssssssssssssssssssssssss  Â
 >                          Â
            |</div>
<div>| 14 Â 9.0 Â 80.6 Â 0.0
 10.4|UUUUssssssssssssssssssssssssssssssssssssssss  Â
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
           |</div>
<div>| 15 Â 15.7 Â 71.1 Â 0.0
 13.2|UUUUUUUsssssssssssssssssssssssssssssssssss  Â
 >|                          Â
            |</div>
<div>| 16 Â 9.2 Â 80.1 Â 0.0
 10.7|UUUUssssssssssssssssssssssssssssssssssssssss  >
 |                            Â
          |</div>
<div>| 17 Â 13.6 Â 74.2 Â 0.0
 12.1|UUUUUUsssssssssssssssssssssssssssssssssssss  >
 |                            Â
          |</div>
<div>| 18 Â 7.2 Â 82.1 Â 0.0
 10.8|UUUsssssssssssssssssssssssssssssssssssssssss > Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 19 Â 16.5 Â 69.5 Â 0.0
 14.0|UUUUUUUUssssssssssssssssssssssssssssssssss  > Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 20 Â 7.3 Â 81.2 Â 0.0
 11.5|UUUssssssssssssssssssssssssssssssssssssssss   >
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 21 Â 17.8 Â 69.0 Â 0.0
 13.2|UUUUUUUUssssssssssssssssssssssssssssssssss   >
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 22 Â 11.4 Â 77.2 Â 0.0
 11.4|UUUUUssssssssssssssssssssssssssssssssssssss > Â
 |                            Â
          |</div>
<div>| 23 Â 14.4 Â 71.1 Â 0.0
 14.4|UUUUUUUsssssssssssssssssssssssssssssssssss  > Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>| 24 Â 5.8 Â 81.7 Â 0.0
 12.6|UUssssssssssssssssssssssssssssssssssssssss >  Â
| Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â
          |</div>
<div>|---------------------------+-------------------------------------------------+
                            Â
         |</div>
<div>|Avg  11.0  77.3  0.0
 11.7|UUUUUssssssssssssssssssssssssssssssssssssss > Â
 |                            Â
          |</div>
<div>|---------------------------+-------------------------------------------------+
 </div>
</div>
<div><br>
</div>
</div>
<br>
<br>
<span title="neteasefooter"><span id="netease_mail_footer"></span></span><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
</body>
</html>