Johnny,<br><br>Thanks for the lesson! I am always happy to learn. Like I said, I am not an expert in TCP/IP.<br><br>What I was writing about when I said that packets are acknowledged is what I saw in Wireshark while trying to understand performance issues. I perhaps should have said "TCP/IP" instead of just "TCP". There were definitely acknowledgements, but I guess they were at the IP level.<br>
<br>I wonder what the MSS is for loopback? I think it's about 1536 on my eth0 interface, but not sure.<br><br>As for RTT, I sent data over a link that had a very long (290ms) RTT, and that definitely limited the rate at which packets could be sent. Can RTT be used to calculate the theoretical maximum traffic that a link can carry? For example, a satellite link with a 400ms RTT but 2 Mbps bandwidth?<br>
<br>Ed<br><br><div class="gmail_quote">On Tue, Jun 24, 2008 at 6:00 PM, Johnny Billquist <<a href="mailto:bqt@softjar.se">bqt@softjar.se</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
No. TCP don't acknowledge every packet. In fact, TCP don't acknowledge packets as such at all. TCP is not packet based. It's just that if you use IP as the carrier, IP itself it packet based.<br>
TCP can in theory generate any number of packets per second. However, the amount of unacknowledged data that can be outstanding at any time is limited by the transmit window. Each packet carries a window size, which is how much more data that can be accepted by the reciever. TCP can (is allowed to) send that much data and no more.<br>
<br>
The RTT calculations are used for figuring out how long to wait before doing retransmissions. You also normally have a slow start transmission algorithm which prevents the sender from even using the full window size from the start, as a way of avoiding congestions. That is used in combination with a backoff algorithm when retransmissions are needed to further decrease congestions, but all of this only really comes into effect if you start loosing data, and TCP actually needs to do retransmissions.<br>
<br>
Another thing you have is an algorithm called Nagle, which tries to collect small amount of data sent into larger packets before sending it, so that you don't flood the net with silly small packets.<br>
<br>
One addisional detail is that receivers normally, when the receive buffers becomes full, don't announce newly freed space immediately, since that is normally rather small amounts, but instead wait a while, until a larger part of the receive buffer is free, so that the sender actually can send some full sized packets once it starts sending again.<br>
<br>
In addition to all this, you also have a max segment size which is negotiated between the TCP ends, which limit the size of a single IP packet sent by the TCP protocol. This is done in order to try to avoid packet fragmentation.<br>
<br>
So the window size is actually a flow control mechanism, and is in reality limiting the amount of data that can be sent. And it varies all the time. And the number of packets that will be used for sending that much data is determined by the MSS (Max Segment Size).<br>
<br>
Sorry for the long text on how TCP works. :-)<br>
<br>
Johnny<br>
<br>
Edwin Fine wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">
David,<br>
<br>
Thanks for trying out the benchmark.<br>
<br></div>
With my limited knowledge of TCP/IP, I believe you are seeing the 300,000 limit because TCP/IP requires acknowledgements to each packet, and although it can batch up multiple acknowledgements in one packet, there is a theoretical limit of packets per seconds beyond which it cannot go due to the laws of physics. I understand that limit is determined by the Round-Trip Time (RTT), which can be shown by ping. On my system, pinging <a href="http://127.0.0.1" target="_blank">127.0.0.1</a> <<a href="http://127.0.0.1" target="_blank">http://127.0.0.1</a>> gives a minimum RTT of 0.018 ms (out of 16 pings). That means that the maximum number of packets that can make it to and dest and back per second is 1/0.000018 seconds, or 55555 packets per second. The TCP/IP stack is evidently packing 5 or 6 blocks into each packet to get the 300K blocks/sec you are seeing. Using Wireshark or Ethereal would confirm this. I am guessing that this means that the TCP window is about 6 * 1000 bytes or 6KB.<div class="Ih2E3d">
<br>
<br>
What I neglected to tell this group is that I have modified the Linux sysctl.conf as follows, which might have had an effect (like I said, I am not an expert):<br>
<br>
# increase Linux autotuning TCP buffer limits<br>
# min, default, and max number of bytes to use<br>
# set max to at least 4MB, or higher if you use very high BDP paths<br>
net.ipv4.tcp_rmem = 4096 87380 16777216<br>
net.ipv4.tcp_wmem = 4096 32768 16777216<br>
<br>
When I have more time, I will vary a number of different Erlang TCP/IP parameters and get a data set together that gives a broader picture of the effect of the parameters.<br>
<br>
Thanks again for taking the time.<br>
<br></div>
2008/6/24 David Mercer <<a href="mailto:dmercer@gmail.com" target="_blank">dmercer@gmail.com</a> <mailto:<a href="mailto:dmercer@gmail.com" target="_blank">dmercer@gmail.com</a>>>:<div class="Ih2E3d"><br>
<br>
I tried some alternative block sizes (using the blksize option). I<br>
found that from 1 to somewhere around––maybe a bit short of––1000<br>
bytes, the test was able to send about 300,000 blocks in 10 seconds<br>
regardless of size. (That means, 0.03 MB/sec for block size of 1,<br>
0.3 MB/sec for block size of 10, 3 MB/sec for block size of 100,<br>
etc.) I suspect the system was CPU bound at those levels.<br>
<br>
<br>
Above 1000, the number of blocks sent seemed to decrease, though<br>
this was more than offset by the increased size of the blocks. Above about 10,000 byte blocks (may have been less, I didn't check<br>
any value between 4,000 and 10,000), however, performance peaked and<br>
block size no longer mattered: it always sent between 70 and 80<br>
MB/sec. My machine is clearly slower than Edwin's…<br>
<br>
<br>
DBM<br>
<br>
<br>
------------------------------------------------------------------------<br>
<br>
*From:* <a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a><br>
<mailto:<a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a>><br>
[mailto:<a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a><br>
<mailto:<a href="mailto:erlang-questions-bounces@erlang.org" target="_blank">erlang-questions-bounces@erlang.org</a>>] *On Behalf Of *Rapsey<br>
*Sent:* Tuesday, June 24, 2008 14:01<br></div>
*To:* <a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><div class="Ih2E3d">
<br>
*Subject:* Re: [erlang-questions] why is gen_tcp:send slow?<br>
<br>
<br>
You're using very large packets. I think the results would be much<br>
more telling if the packets would be a few kB at most. That is<br>
closer to most real life situations.<br>
<br>
<br>
Sergej<br>
<br>
On Tue, Jun 24, 2008 at 8:43 PM, Edwin Fine<br>
<<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a><br></div><div class="Ih2E3d">
<mailto:<a href="mailto:erlang-questions_efine@usa.net" target="_blank">erlang-questions_efine@usa.net</a>>> wrote:<br>
<br>
I wrote a small benchmark in Erlang to see how fast I could get<br>
socket communications to go. All the benchmark does is pump the same<br>
buffer to a socket for (by default) 10 seconds. It uses {active,<br>
once} each time, just like you do.<br>
<br>
Server TCP options:<br>
{active, once},<br>
{reuseaddr, true},<br>
{packet, 0},<br>
{packet_size, 65536},<br>
{recbuf, 1000000}<br>
<br>
Client TCP options:<br>
{packet, raw},<br>
{packet_size, 65536},<br>
{sndbuf, 1024 * 1024},<br>
{send_timeout, 3000}<br>
<br>
Here are some results using Erlang R12B-3 (erl +K true in the Linux<br>
version):<br>
<br>
Linux (Ubuntu 8.10 x86_64, Intel Core 2 Q6600, 8 GB):<br></div>
- Using localhost (<a href="http://127.0.0.1" target="_blank">127.0.0.1</a> <<a href="http://127.0.0.1" target="_blank">http://127.0.0.1</a>>): 7474.14 MB in<div><div></div><div class="Wj3C7c"><br>
10.01 secs (746.66 MB/sec)<br>
- Using 192.168.x.x IP address: 8064.94 MB in 10.00 secs (806.22<br>
MB/sec) [Don't ask me why it's faster than using loopback, I<br>
repeated the tests and got the same result]<br>
<br>
Windows XP SP3 (32 bits), Intel Core 2 Duo E6600:<br>
- Using loopback: 2166.97 MB in 10.02 secs (216.35 MB/sec)<br>
- Using 192.168.x.x IP address: 2140.72 MB in 10.02 secs (213.75 MB/sec)<br>
- On Gigabit Ethernet to the Q6600 Linux box: 1063.61 MB in 10.02<br>
secs (106.17 MB/sec) using non-jumbo frames. I don't think my router<br>
supports jumbo frames.<br>
<br>
There's undoubtedly a huge discrepancy between the two systems,<br>
whether because of kernel poll in Linux, or that it's 64 bits, or<br>
unoptimized Windows TCP/IP flags, I don't know. I don't believe it's<br>
the number of CPUs (there's only 1 process sending and one<br>
receiving), or the CPU speed (they are both 2.4 GHz Core 2s).<br>
<br>
Maybe some Erlang TCP/IP gurus could comment.<br>
<br>
I've attached the code for interest. It's not supposed to be<br>
production quality, so please don't beat me up :) although I am<br>
always open to suggestions for improvement. If you do improve it,<br>
I'd like to see what you've done. Maybe there is another simple<br>
Erlang tcp benchmark program out there (i.e. not Tsung), but I<br>
couldn't find one in a cursory Google search.<br>
<br>
To run:<br>
<br>
VM1:<br>
<br>
tb_server:start(Port, Opts).<br>
tb_server:stop() to stop.<br>
<br>
Port = integer()<br>
Opts = []|[opt()]<br>
opt() = {atom(), term()} (Accepts inet setopts options, too)<br>
<br>
The server prints out the transfer rate (for simplicity).<br>
<br>
VM2:<br>
tb_client(Host, Port, Opts).<br>
<br>
Host = atom()|string() hostname or IP address<br>
Port, Opts as in tb_server<br>
<br>
Runs for 10 seconds, sending a 64K buffer as fast as possible to<br>
Host/Port.<br>
You can change this to 20 seconds (e.g.) by adding the tupls<br>
{time_limit, 20000} to Opts.<br>
You can change buffer size by adding the tuple {blksize, Bytes} to Opts.<br>
<br></div></div>
2008/6/20 Rapsey <<a href="mailto:rapsey@gmail.com" target="_blank">rapsey@gmail.com</a> <mailto:<a href="mailto:rapsey@gmail.com" target="_blank">rapsey@gmail.com</a>>>:<div class="Ih2E3d"><br>
<br>
All data goes through nginx which acts as a proxy. Its CPU<br>
consumption is never over 1%.<br>
<br>
<br>
Sergej<br>
<br>
<br>
On Thu, Jun 19, 2008 at 9:35 PM, Javier París Fernández<br></div><div><div></div><div class="Wj3C7c">
<<a href="mailto:javierparis@udc.es" target="_blank">javierparis@udc.es</a> <mailto:<a href="mailto:javierparis@udc.es" target="_blank">javierparis@udc.es</a>>> wrote:<br>
<br>
<br>
El 19/06/2008, a las 20:06, Rapsey escribió:<br>
<br>
<br>
It loops from another module, that way I can update the code at<br>
any time without disrupting anything.<br>
The packets are generally a few hundred bytes big, except<br>
keyframes which tend to be in the kB range. I haven't tried<br>
looking with wireshark. Still it seems a bit odd that a large<br>
CPU consumption would be the symptom. The traffic is strictly<br>
one way. Either someone is sending the stream or receiving it.<br>
The transmit could of course be written with a passive receive,<br>
but the code would be significantly uglier. I'm sure someone<br>
here knows if setting {active, once} every packet is CPU<br>
intensive or not.<br>
It seems the workings of gen_tcp is quite platform dependent. If<br>
I run the code in windows, sending more than 128 bytes per<br>
gen_tcp call significantly decreases network output.<br>
Oh and I forgot to mention I use R12B-3.<br>
<br>
<br>
Hi,<br>
<br>
Without being an expert.<br>
<br>
200-300 mb/s in small (hundreds of bytes) packets means a *lot* of<br>
system calls if you are doing a gen_tcp:send for each one. If you<br>
buffer 3 packets, you are reducing that by a factor of 3 :). I'd try<br>
to do an small test doing the same thing in C and compare the<br>
results. I think it will also eat a lot of CPU.<br>
<br>
About the proxy CPU... I'm a bit lost about it, but speculating<br>
wildly it is possible that the time spent doing the system calls<br>
that gen_tcp is doing is added to the proxy CPU process.<br>
<br>
Regards.<br>
<br>
<br>
<br>
_______________________________________________<br>
erlang-questions mailing list<br></div></div>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><div class="Ih2E3d">
<br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
erlang-questions mailing list<br></div>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a>><div class="Ih2E3d">
<br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
<br>
<br>
<br>
------------------------------------------------------------------------<br>
<br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://www.erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://www.erlang.org/mailman/listinfo/erlang-questions</a><br>
</div></blockquote>
<br>
<br>
</blockquote></div><br>