John,<br><br>When using the {active, once} option, how much processing was the receiving process doing with the packet before setting {active, once} again? I personally wouldn't use {active, true} in a production environment because of its potential to clog up the system.<br>
<br>In our case, the receiving process does just this.<br><br>handle_info({udp, Socket, _IP, _RemotePortNo, Packet} = Req,<br> #state{stats_instance = StatsInstance,<br> port = _InPortNo,<br> overload = Overload,<br>
type = Type} = State) -><br> inet:setopts(Socket, [{active, once}]),<br> case check_overload(Type, Overload) of<br> allow -><br> spawn(?MODULE, handle_udp_packet, [Req, State]),<br> {noreply, State};<br>
{allow, O2} -><br> spawn(?MODULE, handle_udp_packet, [Req, State]),<br> {noreply, State#state{overload = O2}};<br> {deny, O2} -><br> spawn(?MODULE, reject_udp_packet, [Req, State]),<br> {noreply, State#state{overload = O2}}<br>
end;<br><br>cheers<br>Chandru<br><br><div class="gmail_quote">On 17 April 2012 17:00, John-Paul Bader <span dir="ltr"><<a href="mailto:hukl@berlin.ccc.de">hukl@berlin.ccc.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Quick Update:<br>
<br>
<br>
In our quest to find the potential bottlenecks we started benchmarking {active, false} vs {active, once} vs {active, true}.<br>
<br>
We used a rate limited client and measured how many packets were actually handled on the server. Note that when we say 40000 packets/s the client would send a burst of 40000 packets and then sleep the remaining time of that very second.<br>
<br>
{active, once} was the worst, losing up to 50% packets<br>
{active, false} was much better but still losing up to 25% packets<br>
{active, true} was the clear winner with no packet loss at 100000 packets per second. When using two clients at 100k/s it started losing packets but it was still reasonable given the amount of packets.<br>
<br>
Also the VM stayed below 100% CPU and did not increase memory consumption, staying stable below 23MB<br>
<br>
The server is using active, true and passes the message instantly to another process which does the actual work on another thread.<br>
<br>
Also, we ran these benchmarks on the same machine and sent / received all packets via the loopback interface. The next benchmarks could be run from different machines and also with a more sophisticated client that doesn't send the packets in bursts to see if that changes anything.<br>
<br>
During the tests we played with read_packets and recbuf settings but it appeared to have little or no effect at all but we will continue to play around with that.<br>
<br>
~ John<br>
<br>
Chandru wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi John,<br>
<br><div class="im">
In the steady state, we have about 2000/sec, but in error situations,<br>
we've had peaks of up to 20000 packets / sec.<br>
<br>
cheers<br>
Chandru<br>
<br>
On 16 April 2012 10:31, John-Paul Bader <<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a><br></div><div><div class="h5">
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>>> wrote:<br>
<br>
Hey Chandru,<br>
<br>
how many packets per second did you have to deal with and how big<br>
are they? Just to have something to compare to.<br>
<br>
~ John<br>
<br>
Chandru wrote:<br>
<br>
Hi John,<br>
<br>
Our RADIUS server which handles our data network is written in<br>
Erlang.<br>
We've experimented with various values of recbuf and read_packets<br>
options for the UDP socket. We also use {active, once}. The<br>
receiving<br>
process receives a packet and spawns a new process to handle it.<br>
That is<br>
all it does. The spawned process then executes the rest of the<br>
business<br>
logic.<br>
<br>
That won't be your problem. The problem will be to make sure<br>
your system<br>
is stable while handling all those packets. We use overload<br>
control at<br>
the receiver. You have to pretty much look at the entire<br>
execution path<br>
for each packet and ensure there are no bottlenecks. At that kind of<br>
load, every little bottleneck shows up sooner or later.<br>
<br>
cheers<br>
Chandru<br>
<br>
On 15 April 2012 19:08, John-Paul Bader <<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a><br>
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>><br></div></div><div><div class="h5">
<mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a> <mailto:<a href="mailto:hukl@berlin.ccc.de" target="_blank">hukl@berlin.ccc.de</a>>>> wrote:<br>
<br>
Dear list,<br>
<br>
<br>
I'm currently writing a bittorrent tracker in Erlang. While<br>
a naive<br>
implementation of the protocol is quite easy, there are some<br>
performance related challanges where I could use some help.<br>
<br>
In the first test run as a replacement for a very popular<br>
tracker,<br>
my erlang tracker got about 40k requests per second.<br>
<br>
My initial approach was to initialize the socket in one<br>
process with<br>
{active, once}, handle the message in handle_info with minimal<br>
effort and pass the data asynchronously to a freshly spawned<br>
worker<br>
processes which responds to the clients. After spawning the<br>
process<br>
I'm setting the socket back to {active, once}.<br>
<br>
Now when I switched the erlang tracker live the erlang vm was<br>
topping at 100% CPU load. My guess is that the process<br>
handling the<br>
udp packets from the socket could not keep up. Since I'm<br>
still quite<br>
new to the world of erlang I'd like to know if there are<br>
some best<br>
practices / patterns to handle this massive amount of packets.<br>
<br>
For example using the socket in {active, once} might be too<br>
slow?<br>
Also the response to the clients needs to come from the same<br>
port as<br>
the request was coming in. Is it a problem to use the same<br>
socket<br>
for that? Should I pre-spawn a couple of thousand workers and<br>
dispatch the data from the socket to them rather than<br>
spawning them<br>
on each packet?<br>
<br>
It would be really great if you could give some advice or<br>
point me<br>
into the right directions.<br>
<br>
~ John<br></div></div>
______________________________<u></u>_____________________<div class="im"><br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>><br></div>
<mailto:<a href="mailto:erlang-questions@" target="_blank">erlang-questions@</a>__<a href="http://erlang.org" target="_blank">erl<u></u>ang.org</a><br>
<mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>>><br>
<a href="http://erlang.org/mailman/____listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/____<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a>><br>
<<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a>>><div class="im"><br>
<br>
<br>
______________________________<u></u>___________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a> <mailto:<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@<u></u>erlang.org</a>><br>
<a href="http://erlang.org/mailman/__listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/__<u></u>listinfo/erlang-questions</a><br>
<<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a>><br>
<br>
<br>
</div></blockquote><div class="HOEnZb"><div class="h5">
______________________________<u></u>_________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/<u></u>listinfo/erlang-questions</a><br>
</div></div></blockquote></div><br>