[erlang-questions] How to handle a massive amount of UDP packets?

[John-Paul Bader]

Hey Chandru,

how many packets per second did you have to deal with and how big are 
they? Just to have something to compare to.

~ John

Chandru wrote:
> Hi John,
>
> Our RADIUS server which handles our data network is written in Erlang.
> We've experimented with various values of recbuf and read_packets
> options for the UDP socket. We also use {active, once}. The receiving
> process receives a packet and spawns a new process to handle it. That is
> all it does. The spawned process then executes the rest of the business
> logic.
>
> That won't be your problem. The problem will be to make sure your system
> is stable while handling all those packets. We use overload control at
> the receiver. You have to pretty much look at the entire execution path
> for each packet and ensure there are no bottlenecks. At that kind of
> load, every little bottleneck shows up sooner or later.
>
> cheers
> Chandru
>
> On 15 April 2012 19:08, John-Paul Bader <hukl@REDACTED
> <mailto:hukl@REDACTED>> wrote:
>
>     Dear list,
>
>
>     I'm currently writing a bittorrent tracker in Erlang. While a naive
>     implementation of the protocol is quite easy, there are some
>     performance related challanges where I could use some help.
>
>     In the first test run as a replacement for a very popular tracker,
>     my erlang tracker got about 40k requests per second.
>
>     My initial approach was to initialize the socket in one process with
>     {active, once}, handle the message in handle_info with minimal
>     effort and pass the data asynchronously to a freshly spawned worker
>     processes which responds to the clients. After spawning the process
>     I'm setting the socket back to {active, once}.
>
>     Now when I switched the erlang tracker live the erlang vm was
>     topping at 100% CPU load. My guess is that the process handling the
>     udp packets from the socket could not keep up. Since I'm still quite
>     new to the world of erlang I'd like to know if there are some best
>     practices / patterns to handle this massive amount of packets.
>
>     For example using the socket in {active, once} might be too slow?
>     Also the response to the clients needs to come from the same port as
>     the request was coming in. Is it a problem to use the same socket
>     for that? Should I pre-spawn a couple of thousand workers and
>     dispatch the data from the socket to them rather than spawning them
>     on each packet?
>
>     It would be really great if you could give some advice or point me
>     into the right directions.
>
>     ~ John
>     _________________________________________________
>     erlang-questions mailing list
>     erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>     http://erlang.org/mailman/__listinfo/erlang-questions
>     <http://erlang.org/mailman/listinfo/erlang-questions>
>
>