I can roughly feel it after about 15K~20K. <br><br><div class="gmail_quote">On Wed, Jun 22, 2011 at 11:24 PM, Loïc Hoguin <span dir="ltr"><<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Odd.<br>
<br>
I'll try to run the client you wrote to reproduce whenever I receive the<br>
hardware I'm waiting for. Not sure what could do this though. Do you<br>
have an idea of what's the number of connections before it starts<br>
slowing down?<br>
<div class="im"><br>
On 06/22/2011 05:08 PM, Andy W. Song wrote:<br>
</div><div class="im">> Thanks for the link. I removed the send_interval call and retested but<br>
> it didn't make much of a difference. It still takes about 7 and half<br>
> minutes to create 100K connections. During creation I can see several<br>
> time outs  from client side. And the creation speed gets slower when the<br>
> number of connections gets higher. Although the average CPU usage<br>
> doesn't change much.<br>
><br>
> After creation, with 1.5MB/s traffic, the CPU usage is still about the<br>
> same as before which is 30%.<br>
><br>
> Cheers.<br>
> Andy<br>
><br>
> On Wed, Jun 22, 2011 at 10:34 PM, Loïc Hoguin <<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a><br>
</div><div class="im">> <mailto:<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a>>> wrote:<br>
><br>
> Â Â Memory usage sounds a bit high for cowboy_http_req:compact. But maybe<br>
> Â Â not. Good point though it's already in the examples just like hibernate.<br>
> Â Â My bad.<br>
><br>
> Â Â On the other hand if you have a timer per connection it's definitely not<br>
> Â Â going to scale, timers are very slow so having 100k timers isn't going<br>
> Â Â to work well. Remove the send_interval and it should work a lot better.<br>
><br>
> Â Â For future reference:<br>
> Â Â <a href="http://www.erlang.org/doc/efficiency_guide/commoncaveats.html#id58959" target="_blank">http://www.erlang.org/doc/efficiency_guide/commoncaveats.html#id58959</a><br>
><br>
> Â Â Waiting for yet another comparison, have fun. :)<br>
><br>
> Â Â On 06/22/2011 04:27 PM, Andy W. Song wrote:<br>
> Â Â > I actually didn't patch. I want to keep my change as small as<br>
> Â Â possible.<br>
> Â Â > I will remove that send_interval and test it again. Sorry I didn't pay<br>
> Â Â > much attention about this.<br>
> Â Â ><br>
> Â Â > BTW, how can you tell I used or did not use<br>
> Â Â cowboy_http_req:compact? And<br>
> Â Â > I see hibernate is already the default behavior, isn't it?<br>
> Â Â ><br>
> Â Â > Thanks<br>
> Â Â > Andy<br>
> Â Â ><br>
>   > On Wed, Jun 22, 2011 at 10:06 PM, Loïc Hoguin <<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a><br>
> Â Â <mailto:<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a>><br>
</div><div><div></div><div class="h5">> Â Â > <mailto:<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a> <mailto:<a href="mailto:essen@dev-extend.eu">essen@dev-extend.eu</a>>>> wrote:<br>
> Â Â ><br>
> Â Â > Â Â Hello,<br>
> Â Â ><br>
> Â Â > Â Â Did you fix the many issues in your cowboy application code I<br>
> Â Â reported<br>
> Â Â > Â Â in my previous emails? I see you removed the receive after from<br>
> Â Â > Â Â misultin, did you remove it from cowboy too? You could very<br>
> Â Â well just be<br>
> Â Â > Â Â measuring Erlang's timer modules bad performance there.<br>
> Â Â ><br>
> Â Â > Â Â Also I see you're not using cowboy_http_req:compact and the<br>
> Â Â hibernate<br>
> Â Â > Â Â option for cowboy that is doing it properly (unlike your patch).<br>
> Â Â ><br>
> Â Â > Â Â And again, you should probably come to IRC and ask for advice<br>
> Â Â from the<br>
> Â Â > Â Â other people benchmarking it.<br>
> Â Â ><br>
> Â Â > Â Â Cheers.<br>
> Â Â ><br>
> Â Â > Â Â On 06/22/2011 03:14 PM, Andy W. Song wrote:<br>
> Â Â > Â Â > OK, the competition comes again. I finally wrote a client that<br>
> Â Â > Â Â performs<br>
> Â Â > Â Â > decently. It's here<br>
> Â Â > Â Â <<a href="https://github.com/awsong/Golang-WebSocket-Client" target="_blank">https://github.com/awsong/Golang-WebSocket-Client</a>>.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > I tested my previous Cowboy first( I changed send_interval from<br>
> Â Â > Â Â 1000 to<br>
> Â Â > Â Â > 900000 of line 64 of src/websocket_handler.erl in<br>
> Â Â cowboy_examples ).<br>
> Â Â > Â Â > Here is the version:<br>
> Â Â > Â Â > *Cowboy_examples*: 343d002 Initial commit with a default and<br>
> Â Â websocket<br>
> Â Â > Â Â > handler.<br>
> Â Â > Â Â > *Cowboy*: 5d69825 Add a max_connections transport option,<br>
> Â Â Mon May 9<br>
> Â Â > Â Â > 22:01:31 2011<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > The command line I use on the client is "./conn -c 10000 -n<br>
> Â Â 10". It'll<br>
> Â Â > Â Â > create 100K connections, 10K per IP to the server. It takes<br>
> Â Â about<br>
> Â Â > Â Â 7 and<br>
> Â Â > Â Â > half minutes to finish creating these connections. During the<br>
> Â Â > Â Â creation,<br>
> Â Â > Â Â > server CPU usage is about 110%(dual core), client cpu usage is<br>
> Â Â > Â Â under 20%.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Server memory usage(Cowboy residence memory) :<br>
> Â Â > Â Â > Right after Cowboy starts: 525M<br>
> Â Â > Â Â > After 100K connection established: 1842M<br>
> Â Â > Â Â > After 5M packets passed: 2824M<br>
> Â Â > Â Â > After 9M packets passed: 3231M<br>
> Â Â > Â Â > After 87.7 M packets passed, 5746M<br>
> Â Â > Â Â > [{total,<a href="tel:4155518040" value="+14155518040">4155518040</a> <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a>> <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a><br>
</div></div>> Â Â <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a>>> <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a> <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a>><br>
> Â Â > Â Â <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a> <tel:<a href="tel:4155518040" value="+14155518040">4155518040</a>>>>},<br>
<div><div></div><div class="h5">> Â Â > Â Â > Â {processes,1701318856},<br>
> Â Â > Â Â > Â {processes_used,1701299096},<br>
> Â Â > Â Â > Â {system,2454199184},<br>
> Â Â > Â Â > Â {atom,497457},<br>
> Â Â > Â Â > Â {atom_used,482279},<br>
> Â Â > Â Â > Â {binary,1787851232},<br>
> Â Â > Â Â > Â {code,4041219},<br>
> Â Â > Â Â > Â {ets,41231176}]<br>
> Â Â > Â Â > There seems a memory leak. If I don't stop, the memory usage<br>
> Â Â will keep<br>
> Â Â > Â Â > going higher. 40% CPU, 1.5 MB/s traffic.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > *Latest Cowboy and Cowboy_examples* (same send_interval<br>
> Â Â change). It<br>
> Â Â > Â Â > takes about 7 and half minutes to finish creating 100K<br>
> Â Â connections.<br>
> Â Â > Â Â > During the creation, server CPU usage is about 110%(dual<br>
> Â Â core), client<br>
> Â Â > Â Â > cpu usage is under 20%. After 158.9M packets passed, 1464M<br>
> Â Â residence<br>
> Â Â > Â Â > memory used, 30% CPU, 1.5 MB/s traffic, .<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > *Latest Misultin* from github, with patch (see attachment):<br>
> Â Â > Â Â > It takes about 45s to finish creating 100K connections, during<br>
> Â Â > Â Â creation<br>
> Â Â > Â Â > cpu usage is 120%. After creation, residence memory usage is<br>
> Â Â > Â Â 2684M. And<br>
> Â Â > Â Â > it goes down slowly to 1416M after 40M packets are passed, cpu<br>
> Â Â > Â Â usage is<br>
> Â Â > Â Â > about 50%<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Another thing worth to note is that when I hit CTRL-C on the<br>
> Â Â client<br>
> Â Â > Â Â > side, misultin will quickly destroy the connections and<br>
> Â Â release memory<br>
> Â Â > Â Â > and go back to normal. But Cowboy will consume almost all<br>
> Â Â the cpu time<br>
> Â Â > Â Â > and stay there for a long time (I didn't have the patience<br>
> Â Â to measure<br>
> Â Â > Â Â > how long, so just hit ctrl-c to quick erlang).<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Cheers<br>
> Â Â > Â Â > Andy<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > On Wed, Jun 1, 2011 at 8:26 PM, Andy W. Song<br>
> Â Â <<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â > Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>><br>
> Â Â > Â Â > <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>>>> wrote:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â BTW, the latest version uses less memory too, about 3.2G<br>
> Â Â for 200K<br>
> Â Â > Â Â > Â Â connections.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â On Wed, Jun 1, 2011 at 8:23 PM, Andy W. Song<br>
> Â Â > Â Â <<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>><br>
> Â Â > Â Â > Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>>>> wrote:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Missed one attachment.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â On Wed, Jun 1, 2011 at 8:21 PM, Andy W. Song<br>
> Â Â > Â Â <<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>><br>
> Â Â > Â Â > Â Â Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>><br>
> Â Â <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a> <mailto:<a href="mailto:wsongcn@gmail.com">wsongcn@gmail.com</a>>>>> wrote:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â I tested out the latest Misultin with this<br>
> Â Â change, CPU<br>
> Â Â > Â Â usage<br>
> Â Â > Â Â > Â Â Â Â Â Â is about 2 to 3 percent lower.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â With the attached Misultin package, I can still<br>
> Â Â reproduce<br>
> Â Â > Â Â > Â Â Â Â Â Â the large packet problem.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â With latest version, Misultin doesn't report any<br>
> Â Â > Â Â error. But<br>
> Â Â > Â Â > Â Â Â Â Â Â it doesn't reply to the client either. Seems it<br>
> Â Â just drops<br>
> Â Â > Â Â > Â Â Â Â Â Â the packet. Another attached file is the tcpdump<br>
> Â Â result.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â I will try Mochiweb later just to compare.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â On Wed, Jun 1, 2011 at 12:09 AM, Roberto Ostinelli<br>
> Â Â > Â Â > Â Â Â Â Â Â <<a href="mailto:roberto@widetag.com">roberto@widetag.com</a><br>
> Â Â <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a>> <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a><br>
> Â Â <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a>>><br>
> Â Â > Â Â <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a> <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a>><br>
> Â Â <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a> <mailto:<a href="mailto:roberto@widetag.com">roberto@widetag.com</a>>>>> wrote:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â On May 31, 2011, at 12:54 PM, Andy W. Song<br>
> Â Â wrote:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â Would you mind testing it using my C<br>
> Â Â client? Search<br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â "->len" and give<br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â it a value between 1500 and 2000, hopefully<br>
> Â Â > Â Â you'll see<br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â the result I<br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â saw.<br>
> Â Â > Â Â >><br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â Regards<br>
> Â Â > Â Â >> Â Â Â Â Â Â Â Â Andy<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â Hi Andy,<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â will do that.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â Meanwhile, I've added an undocumented option,<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â {ws_no_header, true} so that headers are not<br>
> Â Â > Â Â duplicated<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â in the record and should use less memory.<br>
> Â Â plus, I've<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â optimized binary code for websockets.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â can you please retry the test so we have a<br>
> Â Â matter of<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â comparison?<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â the only thing you need to change is to add this<br>
> Â Â > Â Â option<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â in misultin initialization, i.e.:<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â % start misultin http server<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â start(Port) -><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â misultin:start_link([<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â Â *{ws_no_header, true},*<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â {port, Port}, {loop, fun(Req) -><br>
> Â Â handle_http(Req,<br>
> Â Â > Â Â Port)<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â end},<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â {ws_loop, fun(Ws) -> handle_websocket(Ws) end},<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â {ws_autoexit, false}<br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â ]).<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â please let me know...<br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â Â Â r.<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â Â Â --<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ---------------------------------------------------------------<br>
>   >   >       有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
>   >   >       苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â Â Â --<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ---------------------------------------------------------------<br>
>   >   >     有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
>   >   >     苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > Â Â --<br>
> Â Â > Â Â ><br>
> Â Â ---------------------------------------------------------------<br>
>   >   >   有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
>   >   >   苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > --<br>
> Â Â > Â Â > ---------------------------------------------------------------<br>
>   >   > 有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
>   >   > 苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â ><br>
> Â Â > Â Â > _______________________________________________<br>
> Â Â > Â Â > erlang-questions mailing list<br>
> Â Â > Â Â > <a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
> Â Â <mailto:<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>><br>
> Â Â <mailto:<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
> Â Â <mailto:<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>>><br>
> Â Â > Â Â > <a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
> Â Â ><br>
> Â Â ><br>
> Â Â > Â Â --<br>
>   >   Loïc Hoguin<br>
> Â Â > Â Â Dev:Extend<br>
> Â Â ><br>
> Â Â ><br>
> Â Â ><br>
> Â Â ><br>
> Â Â > --<br>
> Â Â > ---------------------------------------------------------------<br>
>   > 有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
>   > 苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
> Â Â ><br>
><br>
><br>
> Â Â --<br>
>   Loïc Hoguin<br>
> Â Â Dev:Extend<br>
><br>
><br>
><br>
><br>
> --<br>
> ---------------------------------------------------------------<br>
> 有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š<br>
> 苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´<br>
><br>
<br>
<br>
</div></div><font color="#888888">--<br>
Loïc Hoguin<br>
Dev:Extend<br>
</font></blockquote></div><br><br clear="all"><br>-- <br><div>---------------------------------------------------------------</div><div>有志者，事竟æˆï¼Œç ´é‡œæ²‰èˆŸï¼Œç™¾äºŒç§¦å…³ç»ˆå±žæ¥š</div><div>苦心人，天ä¸è´Ÿï¼Œå§è–ªå°èƒ†ï¼Œä¸‰åƒè¶Šç”²å¯åžå´</div><br>