Hi!<div><br></div><div>I revisited the client code jlouis wrote for the UDP tracker client in etorrent. It turns out that my first assumption was wrong, it does not encode the packets using bencode, instead opting for a much more compact data representation. This may explain why you're not seeing any improvements from my first suggestion.</div>
<div><br></div><div>I'd love to see this open sourced, bittorrent related projects are always interesting, projects aiming for high throughput are also interesting because they are so unforgiving on poor programming practices. Yours is both.</div>
<div><br></div><div>MVH Magnus<br><br><div class="gmail_quote">On Tue, Apr 17, 2012 at 6:00 PM, John-Paul Bader <span dir="ltr"><<a href="mailto:hukl@berlin.ccc.de">hukl@berlin.ccc.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Quick Update:<br>
<br>
<br>
In our quest to find the potential bottlenecks we started benchmarking {active, false} vs {active, once} vs {active, true}.<br>
<br>
We used a rate limited client and measured how many packets were actually handled on the server. Note that when we say 40000 packets/s the client would send a burst of 40000 packets and then sleep the remaining time of that very second.<br>
<br>
{active, once} was the worst, losing up to 50% packets<br>
{active, false} was much better but still losing up to 25% packets<br>
{active, true} was the clear winner with no packet loss at 100000 packets per second. When using two clients at 100k/s it started losing packets but it was still reasonable given the amount of packets.<br>
<br>
Also the VM stayed below 100% CPU and did not increase memory consumption, staying stable below 23MB<br>
<br>
The server is using active, true and passes the message instantly to another process which does the actual work on another thread.<br>
<br>
Also, we ran these benchmarks on the same machine and sent / received all packets via the loopback interface. The next benchmarks could be run from different machines and also with a more sophisticated client that doesn't send the packets in bursts to see if that changes anything.<br>
<br>
During the tests we played with read_packets and recbuf settings but it appeared to have little or no effect at all but we will continue to play around with that.<br>
<br>
~ John<br>
<br>
Chandru wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi John,<br>
<br><div class="im">
In the steady state, we have about 2000/sec, but in error situations,<br>
we've had peaks of up to 20000 packets / sec.<br>
<br>
cheers<br>
Chandru<br>
<br>
On 16 April 2012 10:31, John-Paul Bader <<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a><br></div><div><div class="h5">
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>>> wrote:<br>
<br>
Hey Chandru,<br>
<br>
how many packets per second did you have to deal with and how big<br>
are they? Just to have something to compare to.<br>
<br>
~ John<br>
<br>
Chandru wrote:<br>
<br>
Hi John,<br>
<br>
Our RADIUS server which handles our data network is written in<br>
Erlang.<br>
We've experimented with various values of recbuf and read_packets<br>
options for the UDP socket. We also use {active, once}. The<br>
receiving<br>
process receives a packet and spawns a new process to handle it.<br>
That is<br>
all it does. The spawned process then executes the rest of the<br>
business<br>
logic.<br>
<br>
That won't be your problem. The problem will be to make sure<br>
your system<br>
is stable while handling all those packets. We use overload<br>
control at<br>
the receiver. You have to pretty much look at the entire<br>
execution path<br>
for each packet and ensure there are no bottlenecks. At that kind of<br>
load, every little bottleneck shows up sooner or later.<br>
<br>
cheers<br>
Chandru<br>
<br>
On 15 April 2012 19:08, John-Paul Bader <<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a><br>
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>><br></div></div><div><div class="h5">
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a> <mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>>>> wrote:<br>
<br>
Dear list,<br>
<br>
<br>
I'm currently writing a bittorrent tracker in Erlang. While<br>
a naive<br>
implementation of the protocol is quite easy, there are some<br>
performance related challanges where I could use some help.<br>
<br>
In the first test run as a replacement for a very popular<br>
tracker,<br>
my erlang tracker got about 40k requests per second.<br>
<br>
My initial approach was to initialize the socket in one<br>
process with<br>
{active, once}, handle the message in handle_info with minimal<br>
effort and pass the data asynchronously to a freshly spawned<br>
worker<br>
processes which responds to the clients. After spawning the<br>
process<br>
I'm setting the socket back to {active, once}.<br>
<br>
Now when I switched the erlang tracker live the erlang vm was<br>
topping at 100% CPU load. My guess is that the process<br>
handling the<br>
udp packets from the socket could not keep up. Since I'm<br>
still quite<br>
new to the world of erlang I'd like to know if there are<br>
some best<br>
practices / patterns to handle this massive amount of packets.<br>
<br>
For example using the socket in {active, once} might be too<br>
slow?<br>
Also the response to the clients needs to come from the same<br>
port as<br>
the request was coming in. Is it a problem to use the same<br>
socket<br>
for that? Should I pre-spawn a couple of thousand workers and<br>
dispatch the data from the socket to them rather than<br>
spawning them<br>
on each packet?<br>
<br>
It would be really great if you could give some advice or<br>
point me<br>
into the right directions.<br>
<br>
~ John<br></div></div>
______________________________<u></u>_____________________<div class="im"><br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>><br></div>
<mailto:<a href="mailto:erlang-questions@" target="_blank">erlang-questions@</a>__<a href="http://erlang.org" target="_blank">erl<u></u>ang.org</a><br>
<mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>>><br>
<a href="http://erlang.org/mailman/____listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/____<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a>><br>
<<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a>>><div class="im"><br>
<br>
<br>
______________________________<u></u>___________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>><br>
<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a>><br>
<br>
<br>
</div></blockquote><div class="HOEnZb"><div class="h5">
______________________________<u></u>_________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a><br>
</div></div></blockquote></div><br></div>