<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Thanks for you reply.</div><div><br></div><div>I got more informations from system,there is the CPU analyze result,looks like the most cost is <span style="line-height: 1.7;">sched_yield.What calls in erlang can trigger a </span><span style="line-height: 1.7;">sched_yield?</span></div><div><div>% time seconds usecs/call calls errors syscall</div><div>------ ----------- ----------- --------- --------- ----------------</div><div> 83.95 51.691294 18 2917715 sched_yield</div><div> 13.23 8.147366 60 136096 53085 futex</div><div> 0.92 0.568038 18 31832 4954 recvfrom</div><div> 0.91 0.559971 16 35466 51 writev</div><div> 0.29 0.177944 14 12999 5 stat</div><div> 0.28 0.172864 16 10555 196 epoll_ctl</div><div> 0.16 0.095549 95549 1 vfork</div><div> 0.08 0.048692 73 666 munmap</div><div> 0.07 0.046025 1 77373 74974 close</div><div> 0.07 0.045406 16 2815 epoll_wait</div><div> 0.01 0.008748 35 253 mmap</div><div> 0.01 0.004698 36 132 getdents</div><div> 0.01 0.003173 20 156 getsockopt</div><div> 0.00 0.001910 18 104 setsockopt</div><div> 0.00 0.001204 16 74 getpeername</div><div> 0.00 0.000997 13 75 open</div><div> 0.00 0.000336 8 42 fcntl</div><div> 0.00 0.000136 7 20 1 accept</div><div> 0.00 0.000000 0 1 read</div><div> 0.00 0.000000 0 5 write</div><div> 0.00 0.000000 0 2 fstat</div><div> 0.00 0.000000 0 3 mprotect</div><div> 0.00 0.000000 0 1 brk</div><div> 0.00 0.000000 0 13 1 access</div><div> 0.00 0.000000 0 2 pipe</div><div> 0.00 0.000000 0 2 dup2</div><div> 0.00 0.000000 0 1 execve</div><div> 0.00 0.000000 0 1 getcwd</div><div> 0.00 0.000000 0 1 times</div><div> 0.00 0.000000 0 1 arch_prctl</div><div>------ ----------- ----------- --------- --------- ----------------</div><div>100.00 61.574351 3226407 133267 total</div></div><br><span style="line-height: 1.7;">In erlang node there is a process called '</span>erl_prim_loader',this process has the biggest reduction,is it normall?<br><div>[{registered_name,erl_prim_loader},</div><div> {current_function,{erl_prim_loader,loop,3}},</div><div> {initial_call,{erlang,apply,2}},</div><div> {status,waiting},</div><div> {message_queue_len,0},</div><div> {messages,[]},</div><div> {links,[#Port<0.1>,<0.0.0>]},</div><div> {dictionary,[]},</div><div> {trap_exit,true},</div><div> {error_handler,error_handler},</div><div> {priority,normal},</div><div> {group_leader,<0.0.0>},</div><div> {total_heap_size,57314},</div><div> {heap_size,10946},</div><div> {stack_size,6},</div><div> {reductions,7774178639},</div><div> {garbage_collection,[{min_bin_vheap_size,10946},</div><div> {min_heap_size,10946},</div><div> {fullsweep_after,65535},</div><div> {minor_gcs,12}]},</div><div> {suspending,[]}]</div><br><div></div><div id="divNeteaseMailCard"></div><br>At 2014-12-21 00:53:04, "Guilherme Andrade" <g@gandrade.net> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
Looking into kernel threads (both current CPU utilisation and
consumed CPU time) in something like ps / htop should help.<br>
<br>
Cheers,<br>
<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 16:49, Guilherme Andrade
wrote:<br>
</div>
<blockquote cite="mid:5495A8A5.7040103@gandrade.net" type="cite">
Now that I look more carefully, it looks like your system is
spending a whole lot of CPU time (~75%) in kernel space. Some
system call (or calls) is taking away way too many resources.<br>
<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 16:19, Guilherme
Andrade wrote:<br>
</div>
<blockquote cite="mid:5495A19E.2020405@gandrade.net" type="cite">
Hello gongtao,<br>
<br>
Besides 'message_queue_len', I would also look into 'reductions'
over time (also using process_info/2.)<br>
<br>
As for big message queues, if the CPU is not overloaded, in most
cases it's due to single processes becoming a bottleneck,
whether due to IO and/or heavy crunching; if they can be pooled
or split into multiple ones, the better. However, you say that
CPU is topping at 100%, so it's most likely that it's just too
heavy.<br>
<br>
Long-running NIFs could also cause weird behaviour in the
scheduler.<br>
<br>
This dirty (multiple) one-liner should give you some insight
into reduction count:<br>
<br>
------------------<br>
rp([{-Reduc, Pid, case process_info(Pid, registered_name) of
{registered_name,Name} -> Name; _ -> '_' end} || <br>
{Reduc, Pid} <- <br>
lists:foldl( <br>
fun(Pid, L) when length(L) > 40 -> <br>
SL = lists:sublist(lists:keysort(1, L), 20), <br>
case process_info(Pid, reductions) of <br>
{reductions,Reduc} -> [{-Reduc, Pid} |
SL]; <br>
undefined -> L <br>
end; <br>
(Pid, L) -> <br>
case process_info(Pid, reductions) of<br>
{reductions,Reduc} -> [{-Reduc, Pid} |
L]; <br>
undefined -> L <br>
end <br>
end, [], erlang:processes())]).<br>
------------------<br>
<br>
<br>
Cheers,<br>
<br>
<div class="moz-cite-prefix">On 20-12-2014 12:03, gongtao wrote:<br>
</div>
<blockquote cite="mid:4afb7459.10962.14a67946226.Coremail.goto310@163.com" type="cite">
<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">
<div>hello everyone:</div>
<div> I am a programmer from china.I have a problem with
my game server.There is 8193 processes in my erlang
node.But the cpu usage is nearly full,and the computer
have 24 cpus.I looked some erlang processes info,find the
message_queue_len of process is very big,more than 1000
sometimes.The disk io and net io are normal,memory is
normal.My erlang version is R15B03.I don't know why.</div>
<div> My english is poor.There are same informations from
system.Hope you can understand.</div>
<div> Looking forward to your reply.</div>
<div> Thanks.</div>
<div><br>
</div>
<div>
<div>-bash-4.2$ sar -u 2 10</div>
<div>Linux 2.6.32-279.el6.x86_64 12/20/2014
_x86_64_ (24 CPU)</div>
<div><br>
</div>
<div>06:33:37 PM CPU %user %nice %system
%iowait %steal %idle</div>
<div>06:33:39 PM all 11.21 0.00 75.04
0.00 0.00 13.75</div>
<div>06:33:41 PM all 10.91 0.00 77.22
0.00 0.00 11.86</div>
<div>06:33:43 PM all 12.08 0.00 75.96
0.00 0.00 11.97</div>
<div>06:33:45 PM all 10.50 0.00 77.71
0.00 0.00 11.79</div>
<div>06:33:47 PM all 11.09 0.00 76.75
0.00 0.00 12.16</div>
<div>06:33:49 PM all 11.07 0.00 76.03
0.00 0.00 12.90</div>
<div>06:33:51 PM all 12.07 0.00 75.27
0.00 0.00 12.66</div>
<div>06:33:53 PM all 11.36 0.00 76.84
0.00 0.00 11.80</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>| CPU Utilisation
-------------------------------------------------------------------------------------------------------------------------------------
|</div>
<div>|---------------------------+-------------------------------------------------+
|</div>
<div>|CPU User% Sys% Wait% Idle|0 |25
|50 |75 100|
|</div>
<div>| 1 14.1 75.9 0.0
9.9|UUUUUUUsssssssssssssssssssssssssssssssssssss
>|
|</div>
<div>| 2 9.8 81.3 0.0
8.8|UUUUssssssssssssssssssssssssssssssssssssssss
>|
|</div>
<div>| 3 11.6 76.9 0.0
11.6|UUUUUssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 4 7.6 82.3 0.0
10.1|UUUsssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 5 14.2 74.1 0.0
11.7|UUUUUUUsssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 6 8.6 81.3 0.0
10.1|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 7 11.6 75.9 0.0
12.6|UUUUUsssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 8 8.5 81.0 0.0
10.5|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 9 11.7 76.1 0.0
12.2|UUUUUssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 10 9.1 80.8 0.0
10.1|UUUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 11 10.2 76.5 0.0
13.3|UUUUUssssssssssssssssssssssssssssssssssssss
> |
|</div>
<div>| 12 6.6 81.6 0.0
11.7|UUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 13 12.1 75.3 0.0
12.6|UUUUUUsssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 14 9.0 80.6 0.0
10.4|UUUUssssssssssssssssssssssssssssssssssssssss
>
|</div>
<div>| 15 15.7 71.1 0.0
13.2|UUUUUUUsssssssssssssssssssssssssssssssssss
>|
|</div>
<div>| 16 9.2 80.1 0.0
10.7|UUUUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 17 13.6 74.2 0.0
12.1|UUUUUUsssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 18 7.2 82.1 0.0
10.8|UUUsssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 19 16.5 69.5 0.0
14.0|UUUUUUUUssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 20 7.3 81.2 0.0
11.5|UUUssssssssssssssssssssssssssssssssssssssss
> |
|</div>
<div>| 21 17.8 69.0 0.0
13.2|UUUUUUUUssssssssssssssssssssssssssssssssss
> |
|</div>
<div>| 22 11.4 77.2 0.0
11.4|UUUUUssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 23 14.4 71.1 0.0
14.4|UUUUUUUsssssssssssssssssssssssssssssssssss >
|
|</div>
<div>| 24 5.8 81.7 0.0
12.6|UUssssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>|---------------------------+-------------------------------------------------+
|</div>
<div>|Avg 11.0 77.3 0.0
11.7|UUUUUssssssssssssssssssssssssssssssssssssss >
|
|</div>
<div>|---------------------------+-------------------------------------------------+
</div>
</div>
<div><br>
</div>
</div>
<br>
<br>
<span title="neteasefooter"><span id="netease_mail_footer"></span></span><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Guilherme
<a class="moz-txt-link-freetext" href="https://www.gandrade.net/">https://www.gandrade.net/</a>
PGP: 0x35CB8191 / 1968 5252 3901 B40F ED8A D67A 9330 79B1 35CB 8191
</pre>
</blockquote></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>