<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">For all using <b>ab </b>or<b> siege</b>, I strongly advice you to move to a better tool.<div>The best one for now which support theadings and the fast <b>libev</b> is <b>weighttp</b> (from Lighttpd webserver project):</div><div><br></div><div><b>weighttp</b> use exactly the same <b>ab</b> syntax. So nothing to change in your benchs.</div><div><br></div><div>Hope this help !</div><div><br></div><div><div>###################################################</div><div># Weighttp:</div><div># <a href="http://redmine.lighttpd.net/projects/weighttp/wiki">http://redmine.lighttpd.net/projects/weighttp/wiki</a></div><div>###################################################</div><div><br></div><div>INSTALL</div><div><br></div><div>1. LibEV (<a href="http://software.schmorp.de/pkg/libev.html">http://software.schmorp.de/pkg/libev.html</a>)</div><div> cvs -z3 -d :pserver:anonymous@cvs.schmorp.de/schmorpforge co libev</div><div> cd libev</div><div> aclocal && automake --add-missing && autoconf</div><div> libtoolize --copy --force --ltdl</div><div> sh autogen.sh && ./configure --prefix=/usr && make && make install</div><div><br></div><div>2. Weighttp (<a href="http://redmine.lighttpd.net/projects/weighttp/wiki">http://redmine.lighttpd.net/projects/weighttp/wiki</a>)</div><div> git clone <a href="git://git.lighttpd.net/weighttp">git://git.lighttpd.net/weighttp</a></div><div> cd weighttp</div><div> ./waf configure</div><div> ./waf build</div><div> ./waf install</div><div><br></div><div>Regards,<br>Zabrane</div><div><br></div><div><div>On Jul 12, 2012, at 1:01 PM, Zabrane Mickael wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi,<br><br>Good news. With the new (today) patch:<br><br>old bench: ~70K rps<br>new bench: ~85K rps<br><br>More than 15K rps handled now !!<br>We're not far from the 100K rps ;-)<br><br>Well done Wei.<br><br>Regards,<br>Zabrane<br><br>On Jul 12, 2012, at 11:58 AM, Wei Cao wrote:<br><br><blockquote type="cite">2012/7/12 Zabrane Mickael <<a href="mailto:zabrane3@gmail.com">zabrane3@gmail.com</a>>:<br></blockquote><blockquote type="cite"><blockquote type="cite">Hi Wei,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">We already surpassed the 100krps on an 8-cores machine with our HTTP server<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">(~150K rps).<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Which erlang version did you use to get ~150k rps on 8-cores machine,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">patched or unpatched?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">We reach the 150K on the unpatched version.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">if it was measured on a unpatched erlang<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">version, would you mind measuring it on the patched version and let me<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">know the result?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I didn't yet adapted our code to use VM with your patch.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I'll keep you informed.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Today I found a lock bottleneck through SystemTap, trace-cmd and lcnt,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">after fixing it, ehttpd on my 16-cores can reach 325k rps.<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">RX packets: 326117 TX packets: 326122<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">RX packets: 326845 TX packets: 326859<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">RX packets: 327983 TX packets: 327996<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">RX packets: 326651 TX packets: 326624<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">This is the upper limit of our Gigabit network card, I run ab on three<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">standalone machines to make enough pressure, I posted the fix to<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">github, have a try ~<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">That's simply fantastic. Could you share your bottleneck tracking method?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Any new VM patch to provide?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">through perf top, I see there is a big percentage of time is wasted in<br></blockquote><blockquote type="cite">kernel _spin_lock<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> 1894.00 16.0% _spin_lock<br></blockquote><blockquote type="cite">/usr/lib/debug/lib/modules/2.6.32-131.21.1.tb477.el6.x86_64/vmlinux<br></blockquote><blockquote type="cite"> 566.00 4.8% process_main<br></blockquote><blockquote type="cite">/home/mingsong.cw/erlangpps/lib/erlang/erts-5.10/bin/beam.smp<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">After dumping and doing a statisics of _spin_lock's call stack via<br></blockquote><blockquote type="cite">trace-cmd, I found most of _spin_lock is called by futex_wake, which<br></blockquote><blockquote type="cite">is invoked by pthread mutex.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Finally, I use lcnt to locate all lock collisions in erlang VM, found<br></blockquote><blockquote type="cite">the mutex timeofday is the bottleneck.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> lock<br></blockquote><blockquote type="cite"> location #tries #collisions collisions [%] time<br></blockquote><blockquote type="cite">[us] duration [%]<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> ----- --------- ------- ------------<br></blockquote><blockquote type="cite">--------------- ---------- -------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">timeofday 'beam/erl_time_sup.c':939 895234 551957<br></blockquote><blockquote type="cite">61.6551 3185159 23.5296<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">timeofday 'beam/erl_time_sup.c':971 408006 264498<br></blockquote><blockquote type="cite">64.8270 1473816 10.8874<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">the mutex timeofday is locked each time erts_check_io is invoked to<br></blockquote><blockquote type="cite">"sync the machine's idea of time", erts_check_io is executed hundreds<br></blockquote><blockquote type="cite">of thounds of times per second, so it's locked too much times, hence<br></blockquote><blockquote type="cite">reduce performance.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I solved this problem by moving the sync operation into a standalone<br></blockquote><blockquote type="cite">thread, invoked 1 time per millisecond<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Regards,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Zabrane<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-- <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Best,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Wei Cao<br></blockquote><br><br></div></blockquote></div><div apple-content-edited="true"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div style="font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></div></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div><br><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div>Regards,</div><div>Zabrane</div></span>
</div>
<br></body></html>