I have kernel-poll enabled (osx supports it). Don't have async threads enabled, because I don't know what it does. I'll try it.<br><br><br>Sergej<br><br><br><div class="gmail_quote">On Fri, Jun 20, 2008 at 8:24 AM, Edwin Fine <<a href="mailto:erlang-questions_efine@usa.net">erlang-questions_efine@usa.net</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Which Erlang command-line options are you using? Specifically, are you using -K true and the -A flags? Does OS/X support kernel poll (-K true)? I saw benchmarks where CPU usage without kernel poll was high (60 - 80%), and without it was much lower (5 - 10%).<br>
<br>I wouldn't necessarily agree that "the workings of gen_tcp is quite platform dependent." I would rather guess that TCP/IP stacks, and TCP/IP parameters, are very different across certain operating systems. The default values are often not even close to optimal. There are numerous registry tweaks to improve Windows TCP/IP performance, for example. I am surprised that you are forced to send only 128 bytes at a time or face lower performance in Erlang on Windows. That seems odd indeed. I would be taking looks at default buffer sizes and the registry hacks that are findable on Google, and then experiment.<br>
<br>I was able to improve performance of an application I am working on from 3 message/sec to 70 msgs/sec simply by spawning a function (to gen_tcp:send the data) that was previously being called sequentially. This was because TCP/IP could now pack multiple packets into the same frame, which previously only had one packet in it. The RTT of the link was dreadful (290ms), so this was a bit of a special case but I think the principle remains the same. Transmitting data in fewer packets means fewer system calls, better utilization of available frame space, and less CPU. Plus using -K true and perhaps +A 128 should improve things.<br>
<br>
Give it a try (if you already haven't) and see if it improves things. Also take a look if you will at <a href="http://www-128.ibm.com/developerworks/linux/library/l-hisock.html" target="_blank">Boost socket performance on Linux</a>, which has some interesting information on this topic.<div>
<div></div><div class="Wj3C7c"><br>
<br><div class="gmail_quote">2008/6/19 Rapsey <<a href="mailto:rapsey@gmail.com" target="_blank">rapsey@gmail.com</a>>:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
It loops from another module, that way I can update the code at any time without disrupting anything.<br>The packets are generally a few hundred bytes big, except keyframes which tend to be in the kB range. I haven't tried looking with wireshark. Still it seems a bit odd that a large CPU consumption would be the symptom. The traffic is strictly one way. Either someone is sending the stream or receiving it.<br>
The transmit could of course be written with a passive receive, but the code would be significantly uglier. I'm sure someone here knows if setting {active, once} every packet is CPU intensive or not. <br>It seems the workings of gen_tcp is quite platform dependent. If I run the code in windows, sending more than 128 bytes per gen_tcp call significantly decreases network output.<br>
Oh and I forgot to mention I use R12B-3.<br><br><br>Sergej<div><div></div><div><br><br><div class="gmail_quote">On Thu, Jun 19, 2008 at 7:39 PM, Edwin Fine <<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">How large is each packet? Can multiple packets fit into one TCP window? Have you looked at the TCP/IP wire-level data with Wireshark/Ethereal to see if the packets are being combined at the TCP level? If you see that you are only getting one packet per TCP frame (assuming a packet is much smaller than the window size), you might be falling foul of the Nagle congestion algorithm. The fact that manually buffering your packets improves performance suggests this may be the case. Nagle says that send, send, send is OK, receive, receive, receive is ok, and even send, receive, send, receive is ok, but you get into trouble if sends and receives are mixed asymmetrically on the same socket (e.g. send, send, receive).<br>
<br>Also, I don't understand your transmit_loop. Where is it looping (or am I misunderstanding something)?<br><br>From what I have seen, people writing Erlang TCP/IP code do an {active, once} receive, and when getting the first packet, drop into another loop that does a passive receive until there's no data waiting, then go back into the {active, once} receive. Are you doing this? I am not sure, but I fear that if all your receives are {active, once} it will incur more CPU overhead than the active/passive split. It's hard to know because I can't see enough of your code to know what you are doing overall. Disclaimer: I'm no Erlang or TCP/IP expert.<br>
<br>Hope this helps.<br><br><div class="gmail_quote">2008/6/19 Rapsey <<a href="mailto:rapsey@gmail.com" target="_blank">rapsey@gmail.com</a>>:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div></div><div>
I have a streaming server written in Erlang. When it was pushing 200-300 mb/s the CPU was getting completely hammered. I traced the problem to gen_tcp:send. <br>So instead of sending every audio/video packet with a single gen_tcp:send call, I buffer 3 packets and then send them all at once. CPU consumption dropped dramatically.<br>
On one of the servers I have a simple proxy, the main process that sends packets between the client and some other server looks like this:<br><br>transmit_loop({tcp, Sock, Data}, P) when P#transdat.client == Sock -><br>
gen_tcp:send(P#transdat.server, Data),<br> inet:setopts(P#transdat.client, [{active, once}]),<br> {ok, P};<br>transmit_loop({tcp, Sock, Data}, P) when P#transdat.server == Sock -><br> gen_tcp:send(P#transdat.client, Data),<br>
inet:setopts(P#transdat.server, [{active, once}]),<br> {ok, P};<br>transmit_loop({start, ServerPort}, P) -><br> {ok, Sock} = gen_tcp:connect("<a href="http://127.0.0.1" target="_blank">127.0.0.1</a>", ServerPort, [binary, {active, once}, {packet, 0}]),<br>
{ok, P#transdat{server = Sock}};<br>transmit_loop({tcp_closed, _}, _) -><br> exit(stop).<br><br>The proxy is eating more CPU time than the streaming server.<br>Is this normal behavior? The server is running OSX 10.4<br>
<br><br>Sergej<br>
<br></div></div>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></div><br>
</blockquote></div><br>
</div></div><br>_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br></blockquote></div><br>
</div></div></blockquote></div><br>