<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div><div>On Jul 10, 2012, at 4:30 AM, Wei Cao wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>My tests were all keepalived/persistent connections, and there was<br>significant performance gain in these situations.</div></blockquote><div><br></div><div>Of course I enabled <b>keepalive</b> on my test.</div><div><br></div><blockquote type="cite"><div>but I wrote a http server with short connection just now, the performance did drop down,<br>I'll find it out later :).<br></div></blockquote><div><br></div>Great. Let me know if you make any progress.</div><div><br></div><div><div>Regards,</div><div>Zabrane</div><div><br></div><blockquote type="cite"><div><br>2012/7/10 Zabrane Mickael <<a href="mailto:zabrane3@gmail.com">zabrane3@gmail.com</a>>:<br><blockquote type="cite">Hi,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Performance of our HTTP Web Server drops down after applying your patches.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Box: Linux F17, 4GB of RAM:<br></blockquote><blockquote type="cite">$ lscpu<br></blockquote><blockquote type="cite">Architecture: i686<br></blockquote><blockquote type="cite">CPU op-mode(s): 32-bit, 64-bit<br></blockquote><blockquote type="cite">Byte Order: Little Endian<br></blockquote><blockquote type="cite">CPU(s): 4<br></blockquote><blockquote type="cite">On-line CPU(s) list: 0-3<br></blockquote><blockquote type="cite">Thread(s) per core: 1<br></blockquote><blockquote type="cite">Core(s) per socket: 4<br></blockquote><blockquote type="cite">Socket(s): 1<br></blockquote><blockquote type="cite">Vendor ID: GenuineIntel<br></blockquote><blockquote type="cite">CPU family: 6<br></blockquote><blockquote type="cite">Model: 23<br></blockquote><blockquote type="cite">Stepping: 7<br></blockquote><blockquote type="cite">CPU MHz: 2499.772<br></blockquote><blockquote type="cite">BogoMIPS: 4999.54<br></blockquote><blockquote type="cite">Virtualization: VT-x<br></blockquote><blockquote type="cite">L1d cache: 32K<br></blockquote><blockquote type="cite">L1i cache: 32K<br></blockquote><blockquote type="cite">L2 cache: 3072K<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bench:<br></blockquote><blockquote type="cite">before: 77787 rps<br></blockquote><blockquote type="cite">after: 53056 rps<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Any hint to explain this result Wei ?<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Regards,<br></blockquote><blockquote type="cite">Zabrane<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Jul 9, 2012, at 11:01 AM, Wei Cao wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi, all<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">We wrote a proxy server in Erlang, the proxy's logic is rather simple,<br></blockquote><blockquote type="cite">it listens on some TCP port, establishes new connection from user<br></blockquote><blockquote type="cite">client, forward packets back and forth between the client and backend<br></blockquote><blockquote type="cite">server after authentication until connection is closed.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">It's very easy to write such a proxy in Erlang, fork a process for<br></blockquote><blockquote type="cite">each new user connection and connect to the backend server in the same<br></blockquote><blockquote type="cite">process, the process works like a pipe, sockets from both side is set<br></blockquote><blockquote type="cite">to the active once mode, whenever a tcp packet is received from one<br></blockquote><blockquote type="cite">socket, the packet will be sent to other socket. (A simplified version<br></blockquote><blockquote type="cite">of proxy code is attached at the end of the mail)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">However, the performance is not quite satisfying, the proxy can handle<br></blockquote><blockquote type="cite">maximum only 40k requests on our 16 core machine(Intel Xeon L5630,<br></blockquote><blockquote type="cite">2.13GHz) with heavy stress(100 concurrent clients). We then analyzed<br></blockquote><blockquote type="cite">the behavior of beam.smp use tools like tcprstat, mpstat, perf, and<br></blockquote><blockquote type="cite">SystemTap.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">tcprstat shows QPS is about 40k, have average 1.7 millisecond latency<br></blockquote><blockquote type="cite">for each request.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg<br></blockquote><blockquote type="cite">99_std<br></blockquote><blockquote type="cite">1341813326 39416 17873 953 1718 1519 724 2919 1609 340 3813 1674 462<br></blockquote><blockquote type="cite">1341813327 40528 9275 884 1645 1493 491 2777 1559 290 3508 1619 409<br></blockquote><blockquote type="cite">1341813328 40009 18105 925 1694 1507 714 2868 1586 328 3753 1650 450<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">mpstat shows 30% CPU is idle,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">03:30:19 PM CPU %usr %nice %sys %iowait %irq %soft<br></blockquote><blockquote type="cite">%steal %guest %idle<br></blockquote><blockquote type="cite">03:30:20 PM all 38.69 0.00 21.92 0.00 0.06 7.52<br></blockquote><blockquote type="cite">0.00 0.00 31.80<br></blockquote><blockquote type="cite">03:30:21 PM all 37.56 0.00 21.99 0.00 0.00 7.50<br></blockquote><blockquote type="cite">0.00 0.00 32.95<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">and perf top shows, much time is wasted in scheduler_wait, in spin wait I<br></blockquote><blockquote type="cite">guess.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">9320.00 19.8% scheduler_wait<br></blockquote><blockquote type="cite">/home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp<br></blockquote><blockquote type="cite">1813.00 3.9% process_main<br></blockquote><blockquote type="cite">/home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp<br></blockquote><blockquote type="cite">1379.00 2.9% _spin_lock<br></blockquote><blockquote type="cite">/usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux<br></blockquote><blockquote type="cite">1201.00 2.6% schedule<br></blockquote><blockquote type="cite">/home/mingsong.cw/erlangr16b/lib/erlang/erts-5.10/bin/beam.smp<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">It seems the performance may be associated with scheduler_wait() and<br></blockquote><blockquote type="cite">erts_check_io(), with a SystemTap script(attached at the end of this<br></blockquote><blockquote type="cite">mail), we can find out how many times the system call epoll_wait is<br></blockquote><blockquote type="cite">invoked by beam.smp and each time, how many revents it gets.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">cpu process times<br></blockquote><blockquote type="cite">revents min max avg timeouts<br></blockquote><blockquote type="cite">all<br></blockquote><blockquote type="cite">1754 128042 - - 73 3<br></blockquote><blockquote type="cite">[14] beam.smp 151<br></blockquote><blockquote type="cite">14127 82 97 93 0<br></blockquote><blockquote type="cite">[ 5] beam.smp 142<br></blockquote><blockquote type="cite">13291 83 97 93 0<br></blockquote><blockquote type="cite">[13] beam.smp 127<br></blockquote><blockquote type="cite">11948 86 96 94 0<br></blockquote><blockquote type="cite">[ 6] beam.smp 127<br></blockquote><blockquote type="cite">11836 81 96 93 0<br></blockquote><blockquote type="cite">[ 4] beam.smp 121<br></blockquote><blockquote type="cite">11323 81 96 93 0<br></blockquote><blockquote type="cite">[15] beam.smp 117<br></blockquote><blockquote type="cite">10935 83 96 93 0<br></blockquote><blockquote type="cite">[12] beam.smp 486<br></blockquote><blockquote type="cite">10128 0 96 20 2<br></blockquote><blockquote type="cite">[ 1] beam.smp 71<br></blockquote><blockquote type="cite">6549 71 100 92 0<br></blockquote><blockquote type="cite">[ 2] beam.smp 62<br></blockquote><blockquote type="cite">5695 82 96 91 0<br></blockquote><blockquote type="cite">[ 7] beam.smp 55<br></blockquote><blockquote type="cite">5102 81 95 92 0<br></blockquote><blockquote type="cite">[11] beam.smp 52<br></blockquote><blockquote type="cite">4822 85 95 92 0<br></blockquote><blockquote type="cite">[ 9] beam.smp 52<br></blockquote><blockquote type="cite">4799 85 95 92 0<br></blockquote><blockquote type="cite">[ 8] beam.smp 51<br></blockquote><blockquote type="cite">4680 78 95 91 0<br></blockquote><blockquote type="cite">[10] beam.smp 49<br></blockquote><blockquote type="cite">4508 85 97 92 0<br></blockquote><blockquote type="cite">[ 3] beam.smp 46<br></blockquote><blockquote type="cite">4211 81 95 91 0<br></blockquote><blockquote type="cite">[ 0] beam.smp 44<br></blockquote><blockquote type="cite">4088 83 95 92 0<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The resuls shows, epoll_wait is invoked 1754 times each second, and<br></blockquote><blockquote type="cite">get 73 io events in average. This is unacceptable for writing high<br></blockquote><blockquote type="cite">performance server. Because if epoll_wait is invoked no more than 2k<br></blockquote><blockquote type="cite">times per second, then read/write a packet would cost more than 500ms,<br></blockquote><blockquote type="cite">which causes long delay and affects the throughput finally.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The problem relies on there is only one global pollset in system wide,<br></blockquote><blockquote type="cite">so at a time there is no more than one scheduler can call<br></blockquote><blockquote type="cite">erts_check_io() to obtain pending io tasks from underlying pollset,<br></blockquote><blockquote type="cite">and no scheduler can call erts_check_io() before all pending io<br></blockquote><blockquote type="cite">tasks're processed, so for IO bounded application, it's very likely<br></blockquote><blockquote type="cite">that a scheduler finish its own job, but must wait idly for other<br></blockquote><blockquote type="cite">schedulers to complete theirs.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hence, we develops a patch to slove this problem, by having a pollset<br></blockquote><blockquote type="cite">for each scheduler, so that each scheduler can invoke erts_check_io()<br></blockquote><blockquote type="cite">on its own pollset concurrently. After a scheduler complete its tasks,<br></blockquote><blockquote type="cite">it can invoke erts_check_io() immediately no matter what state other<br></blockquote><blockquote type="cite">schedulers're in. This patch also handles port migration situation,<br></blockquote><blockquote type="cite">all used file descriptors in each port're recorded, when a port is<br></blockquote><blockquote type="cite">migrated, these<br></blockquote><blockquote type="cite">fd 're removed from original scheduler's pollset, and added to new<br></blockquote><blockquote type="cite">scheduler's.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bind port to scheduler together with process is also helpful to<br></blockquote><blockquote type="cite">performance, it reduces the cost of thread switches and<br></blockquote><blockquote type="cite">synchronization, and bound port won't be migrated between schedulers.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">After apply the two patches, with the same pressure(100 concurrent<br></blockquote><blockquote type="cite">clients),epoll_wait is invoked 49332 times per second, and get 3<br></blockquote><blockquote type="cite">revents each time in average, that is to say, our server responds<br></blockquote><blockquote type="cite">quicker and become more realtime.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">cpu process times<br></blockquote><blockquote type="cite">revents min max avg timeouts<br></blockquote><blockquote type="cite">all<br></blockquote><blockquote type="cite">49332 217186 - - 4 3<br></blockquote><blockquote type="cite">[ 2] beam.smp 3219<br></blockquote><blockquote type="cite">16050 2 7 4 0<br></blockquote><blockquote type="cite">[11] beam.smp 4275<br></blockquote><blockquote type="cite">16033 1 6 3 0<br></blockquote><blockquote type="cite">[ 8] beam.smp 4240<br></blockquote><blockquote type="cite">15992 1 6 3 0<br></blockquote><blockquote type="cite">[ 9] beam.smp 4316<br></blockquote><blockquote type="cite">15964 0 6 3 2<br></blockquote><blockquote type="cite">[10] beam.smp 4139<br></blockquote><blockquote type="cite">15851 1 6 3 0<br></blockquote><blockquote type="cite">[ 3] beam.smp 4256<br></blockquote><blockquote type="cite">15816 1 6 3 0<br></blockquote><blockquote type="cite">[ 1] beam.smp 3107<br></blockquote><blockquote type="cite">15800 2 7 5 0<br></blockquote><blockquote type="cite">[ 0] beam.smp 3727<br></blockquote><blockquote type="cite">15259 1 6 4 0<br></blockquote><blockquote type="cite">[ 7] beam.smp 2810<br></blockquote><blockquote type="cite">14722 3 7 5 0<br></blockquote><blockquote type="cite">[13] beam.smp 1981<br></blockquote><blockquote type="cite">11499 4 7 5 0<br></blockquote><blockquote type="cite">[15] beam.smp 2285<br></blockquote><blockquote type="cite">10942 3 6 4 0<br></blockquote><blockquote type="cite">[14] beam.smp 2258<br></blockquote><blockquote type="cite">10866 3 6 4 0<br></blockquote><blockquote type="cite">[ 4] beam.smp 2246<br></blockquote><blockquote type="cite">10849 3 6 4 0<br></blockquote><blockquote type="cite">[ 6] beam.smp 2206<br></blockquote><blockquote type="cite">10730 3 6 4 0<br></blockquote><blockquote type="cite">[12] beam.smp 2173<br></blockquote><blockquote type="cite">10573 3 6 4 0<br></blockquote><blockquote type="cite">[ 5] beam.smp 2093<br></blockquote><blockquote type="cite">10240 3 6 4 0<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">scheduler_wait no longer take so much time now,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> 169.00 6.2% process_main beam.smp<br></blockquote><blockquote type="cite"> 55.00 2.0% _spin_lock [kernel]<br></blockquote><blockquote type="cite"> 45.00 1.7% driver_deliver_term beam.smp<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">so is idle CPU time<br></blockquote><blockquote type="cite">04:30:44 PM CPU %usr %nice %sys %iowait %irq %soft<br></blockquote><blockquote type="cite">%steal %guest %idle<br></blockquote><blockquote type="cite">04:30:45 PM all 60.34 0.00 21.44 0.00 0.06 16.45<br></blockquote><blockquote type="cite">0.00 0.00 1.71<br></blockquote><blockquote type="cite">04:30:46 PM all 60.99 0.00 21.22 0.00 0.00 16.26<br></blockquote><blockquote type="cite">0.00 0.00 1.52<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">and tcprstat shows, QPS is getting 100K, latency is less than 1 millisecond<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">timestamp count max min avg med stddev 95_max 95_avg 95_std 99_max 99_avg<br></blockquote><blockquote type="cite">99_std<br></blockquote><blockquote type="cite">1341822689 96078 11592 314 910 817 311 1447 869 228 1777 897 263<br></blockquote><blockquote type="cite">1341822690 100651 24245 209 914 819 381 1458 870 229 1800 899 263<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I also write a extreamly simple keep-alive http server(attached at the<br></blockquote><blockquote type="cite">end of mail), to compare performance before and after applying the<br></blockquote><blockquote type="cite">patches, mearused with apache ab tool(ab -n 1000000 -c 100 -k ), a 30%<br></blockquote><blockquote type="cite">performance gain can be get.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">before<br></blockquote><blockquote type="cite">Requests per second: 103671.70 [#/sec] (mean)<br></blockquote><blockquote type="cite">Time per request: 0.965 [ms] (mean)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">after<br></blockquote><blockquote type="cite">Requests per second: 133701.24 [#/sec] (mean)<br></blockquote><blockquote type="cite">Time per request: 0.748 [ms] (mean)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Patches can be found at github, compile with<br></blockquote><blockquote type="cite">./configure CFLAGS=-DERTS_POLLSET_PER_SCHEDULER<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Pollset per scheduler:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">git fetch <a href="git://github.com/weicao/otp.git">git://github.com/weicao/otp.git</a> pollset_per_scheduler<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler">https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler</a><br></blockquote><blockquote type="cite"><a href="https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch">https://github.com/weicao/otp/compare/weicao:master...weicao:pollset_per_scheduler.patch</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Bind port to scheduler:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">git fetch <a href="git://github.com/weicao/otp.git">git://github.com/weicao/otp.git</a> bind_port_to_scheduler<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><a href="https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler">https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler</a><br></blockquote><blockquote type="cite"><a href="https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch">https://github.com/weicao/otp/compare/weicao:pollset_per_scheduler...weicao:bind_port_to_scheduler.patch</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Appendix:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-----------------------------------<br></blockquote><blockquote type="cite">proxy.erl<br></blockquote><blockquote type="cite">------------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-module(proxy).<br></blockquote><blockquote type="cite">-compile(export_all).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-define(RECBUF_SIZE, 8192).<br></blockquote><blockquote type="cite">-define(ACCEPT_TIMEOUT, 2000).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">start([MyPortAtom, DestIpAtom, DestPortAtom]) -><br></blockquote><blockquote type="cite"> {MyPort, []} = string:to_integer(atom_to_list(MyPortAtom)),<br></blockquote><blockquote type="cite"> DestIp = atom_to_list(DestIpAtom),<br></blockquote><blockquote type="cite"> {DestPort, []} = string:to_integer(atom_to_list(DestPortAtom)),<br></blockquote><blockquote type="cite"> listen(MyPort, DestIp, DestPort),<br></blockquote><blockquote type="cite"> receive Any -><br></blockquote><blockquote type="cite"> io:format("recv ~p~n", [Any])<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">listen(MyPort, DestIp, DestPort) -><br></blockquote><blockquote type="cite"> io:format("start proxy on [local] 0.0.0.0:~p -> [remote] ~p:~p~n",<br></blockquote><blockquote type="cite">[MyPort, DestIp, DestPort]),<br></blockquote><blockquote type="cite"> case gen_tcp:listen(MyPort,<br></blockquote><blockquote type="cite"> [inet,<br></blockquote><blockquote type="cite"> {ip, {0,0,0,0}},<br></blockquote><blockquote type="cite"> binary,<br></blockquote><blockquote type="cite"> {reuseaddr, true},<br></blockquote><blockquote type="cite"> {recbuf, ?RECBUF_SIZE},<br></blockquote><blockquote type="cite"> {active, false},<br></blockquote><blockquote type="cite"> {nodelay, true}<br></blockquote><blockquote type="cite"> ]) of<br></blockquote><blockquote type="cite"> {ok, Listen} -><br></blockquote><blockquote type="cite"> N = erlang:system_info(schedulers),<br></blockquote><blockquote type="cite"> lists:foreach(fun(I) -> accept(Listen, DestIp, DestPort,<br></blockquote><blockquote type="cite">I) end, lists:seq(1,N));<br></blockquote><blockquote type="cite"> {error, Reason} -><br></blockquote><blockquote type="cite"> io:format("error listen ~p~n", [Reason])<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">accept(Listen, DestIp, DestPort, I) -><br></blockquote><blockquote type="cite"> spawn_opt(?MODULE, loop, [Listen, DestIp, DestPort, I], [{scheduler,<br></blockquote><blockquote type="cite">I}]).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">loop(Listen, DestIp, DestPort, I) -><br></blockquote><blockquote type="cite"> case gen_tcp:accept(Listen, ?ACCEPT_TIMEOUT) of<br></blockquote><blockquote type="cite"> {ok, S1} -><br></blockquote><blockquote type="cite"> accept(Listen, DestIp, DestPort, I),<br></blockquote><blockquote type="cite"> case catch gen_tcp:connect(DestIp, DestPort,<br></blockquote><blockquote type="cite"> [inet, binary, {active, false},<br></blockquote><blockquote type="cite"> {reuseaddr, true}, {nodelay, true}]) of<br></blockquote><blockquote type="cite"> {ok, S2} -><br></blockquote><blockquote type="cite"> io:format("new connection~n"),<br></blockquote><blockquote type="cite"> loop1(S1, S2);<br></blockquote><blockquote type="cite"> {error, Reason} -><br></blockquote><blockquote type="cite"> io:format("error connect ~p~n", [Reason])<br></blockquote><blockquote type="cite"> end;<br></blockquote><blockquote type="cite"> {error, timeout} -><br></blockquote><blockquote type="cite"> loop(Listen, DestIp, DestPort, I);<br></blockquote><blockquote type="cite"> Error -><br></blockquote><blockquote type="cite"> io:format("error accept ~p~n", [Error]),<br></blockquote><blockquote type="cite"> accept(Listen, DestIp, DestPort, I)<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">loop1(S1, S2) -><br></blockquote><blockquote type="cite"> active(S1, S2),<br></blockquote><blockquote type="cite"> receive<br></blockquote><blockquote type="cite"> {tcp, S1, Data} -><br></blockquote><blockquote type="cite"> gen_tcp:send(S2, Data),<br></blockquote><blockquote type="cite"> loop1(S1, S2);<br></blockquote><blockquote type="cite"> {tcp, S2, Data} -><br></blockquote><blockquote type="cite"> gen_tcp:send(S1, Data),<br></blockquote><blockquote type="cite"> loop1(S1, S2);<br></blockquote><blockquote type="cite"> {tcp_closed, S1} -><br></blockquote><blockquote type="cite"> io:format("S1 close~n"),<br></blockquote><blockquote type="cite"> gen_tcp:close(S1),<br></blockquote><blockquote type="cite"> gen_tcp:close(S2);<br></blockquote><blockquote type="cite"> {tcp_closed, S2} -><br></blockquote><blockquote type="cite"> io:format("S2 close~n"),<br></blockquote><blockquote type="cite"> gen_tcp:close(S1),<br></blockquote><blockquote type="cite"> gen_tcp:close(S2)<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">active(S1,S2)-><br></blockquote><blockquote type="cite"> inet:setopts(S1, [{active, once}]),<br></blockquote><blockquote type="cite"> inet:setopts(S2, [{active, once}]).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-----------------------------------<br></blockquote><blockquote type="cite">epollwait.stp<br></blockquote><blockquote type="cite">------------------------------------<br></blockquote><blockquote type="cite">#! /usr/bin/env stap<br></blockquote><blockquote type="cite">#<br></blockquote><blockquote type="cite">#<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">global epoll_timeout_flag, epoll_count, epoll_min, epoll_max,<br></blockquote><blockquote type="cite">epoll_times, epoll_timeouts<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">probe syscall.epoll_wait {<br></blockquote><blockquote type="cite"> if(timeout > 0) {<br></blockquote><blockquote type="cite"> epoll_timeout_flag[pid()] = 1<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite">}<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">probe syscall.epoll_wait.return {<br></blockquote><blockquote type="cite"> c = cpu()<br></blockquote><blockquote type="cite"> p = execname()<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> epoll_times[c,p] ++<br></blockquote><blockquote type="cite"> epoll_count[c,p] += $return<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> if($return == 0 && pid() in epoll_timeout_flag) {<br></blockquote><blockquote type="cite"> epoll_timeouts[c,p] ++<br></blockquote><blockquote type="cite"> delete epoll_timeout_flag[pid()]<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> if(!([c, p] in epoll_min)) {<br></blockquote><blockquote type="cite"> epoll_min[c,p] = $return<br></blockquote><blockquote type="cite"> } else if($return < epoll_min[c,p]) {<br></blockquote><blockquote type="cite"> epoll_min[c,p] = $return<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> if($return > epoll_max[c,p]) {<br></blockquote><blockquote type="cite"> epoll_max[c,p] = $return<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite">}<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">probe timer.s($1) {<br></blockquote><blockquote type="cite"> printf ("%4s %45s %10s %10s %10s %10s %10s %10s\n", "cpu",<br></blockquote><blockquote type="cite">"process", "times", "revents", "min", "max", "avg", "timeouts" )<br></blockquote><blockquote type="cite"> foreach ([cpu, process] in epoll_count- limit 20) {<br></blockquote><blockquote type="cite"> all_epoll_times += epoll_times[cpu,process]<br></blockquote><blockquote type="cite"> all_epoll_count += epoll_count[cpu,process]<br></blockquote><blockquote type="cite"> all_epoll_timeouts += epoll_timeouts[cpu,process]<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite"> printf ("%4s %45s %10d %10d %10s %10s %10d %10d\n",<br></blockquote><blockquote type="cite"> "all", "", all_epoll_times, all_epoll_count, "-", "-",<br></blockquote><blockquote type="cite"> all_epoll_count == 0? 0:all_epoll_count/all_epoll_times,<br></blockquote><blockquote type="cite">all_epoll_timeouts)<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> foreach ([cpu, process] in epoll_count- limit 20) {<br></blockquote><blockquote type="cite"> printf ("[%2d] %45s %10d %10d %10d %10d %10d %10d\n",<br></blockquote><blockquote type="cite"> cpu, process, epoll_times[cpu, process], epoll_count[cpu, process],<br></blockquote><blockquote type="cite"> epoll_min[cpu, process], epoll_max[cpu, process],<br></blockquote><blockquote type="cite"> epoll_count[cpu,process]/epoll_times[cpu,process],<br></blockquote><blockquote type="cite"> epoll_timeouts[cpu,process])<br></blockquote><blockquote type="cite"> }<br></blockquote><blockquote type="cite"> delete epoll_count<br></blockquote><blockquote type="cite"> delete epoll_min<br></blockquote><blockquote type="cite"> delete epoll_max<br></blockquote><blockquote type="cite"> delete epoll_times<br></blockquote><blockquote type="cite"> delete epoll_timeouts<br></blockquote><blockquote type="cite"> printf<br></blockquote><blockquote type="cite">("--------------------------------------------------------------------------\n\n"<br></blockquote><blockquote type="cite">)<br></blockquote><blockquote type="cite">}<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">------------------------------------------------<br></blockquote><blockquote type="cite">ehttpd.erl<br></blockquote><blockquote type="cite">-------------------------------------------------<br></blockquote><blockquote type="cite">-module(ehttpd).<br></blockquote><blockquote type="cite">-compile(export_all).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">start() -><br></blockquote><blockquote type="cite"> start(8888).<br></blockquote><blockquote type="cite">start(Port) -><br></blockquote><blockquote type="cite"> N = erlang:system_info(schedulers),<br></blockquote><blockquote type="cite"> listen(Port, N),<br></blockquote><blockquote type="cite"> io:format("ehttpd ready with ~b schedulers on port ~b~n", [N, Port]),<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> register(?MODULE, self()),<br></blockquote><blockquote type="cite"> receive Any -> io:format("~p~n", [Any]) end. %% to stop: ehttpd!stop.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">listen(Port, N) -><br></blockquote><blockquote type="cite"> Opts = [inet,<br></blockquote><blockquote type="cite"> binary,<br></blockquote><blockquote type="cite"> {active, false},<br></blockquote><blockquote type="cite"> {recbuf, 8192},<br></blockquote><blockquote type="cite"> {nodelay,true},<br></blockquote><blockquote type="cite"> {reuseaddr, true}],<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> {ok, S} = gen_tcp:listen(Port, Opts),<br></blockquote><blockquote type="cite"> lists:foreach(fun(I)-> spawn_opt(?MODULE, accept, [S, I],<br></blockquote><blockquote type="cite">[{scheduler, I}]) end, lists:seq(1, N)).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">accept(S, I) -><br></blockquote><blockquote type="cite"> case gen_tcp:accept(S) of<br></blockquote><blockquote type="cite"> {ok, Socket} -><br></blockquote><blockquote type="cite"> spawn_opt(?MODULE, accept, [S, I],[{scheduler,I}] ),<br></blockquote><blockquote type="cite"> io:format("new connection @~p~n", [I]),<br></blockquote><blockquote type="cite"> loop(Socket,<<>>);<br></blockquote><blockquote type="cite"> Error -> erlang:error(Error)<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">loop(S,B) -><br></blockquote><blockquote type="cite"> inet:setopts(S, [{active, once}]),<br></blockquote><blockquote type="cite"> receive<br></blockquote><blockquote type="cite"> {tcp, S, Data} -><br></blockquote><blockquote type="cite"> B1 = <<B/binary, Data/binary>>,<br></blockquote><blockquote type="cite"> case binary:part(B1,{byte_size(B1), -4}) of<br></blockquote><blockquote type="cite"> <<"\r\n\r\n">> -><br></blockquote><blockquote type="cite"> Response = <<"HTTP/1.1 200 OK\r\nContent-Length:<br></blockquote><blockquote type="cite">12\r\nConnection: Keep-Alive\r\n\r\nhello world!">>,<br></blockquote><blockquote type="cite"> gen_tcp:send(S, Response),<br></blockquote><blockquote type="cite"> loop(S, <<>>);<br></blockquote><blockquote type="cite"> _ -><br></blockquote><blockquote type="cite"> loop(S, B1)<br></blockquote><blockquote type="cite"> end;<br></blockquote><blockquote type="cite"> {tcp_closed, S} -><br></blockquote><blockquote type="cite"> io:format("connection closed forced~n"),<br></blockquote><blockquote type="cite"> gen_tcp:close(S);<br></blockquote><blockquote type="cite"> Error -><br></blockquote><blockquote type="cite"> io:format("unexpected message~p~n", [Error]),<br></blockquote><blockquote type="cite"> Error<br></blockquote><blockquote type="cite"> end.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Best,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Wei Cao<br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">erlang-questions mailing list<br></blockquote><blockquote type="cite"><a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br></blockquote><blockquote type="cite"><a href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><br><br><br>-- <br><br>Best,<br><br>Wei Cao<br></div></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><br></div></span>
</div>
<br></div></div></body></html>