[erlang-questions] How to handle a massive amount of UDP packets?

[Chandru]

Hi John,

In the steady state, we have about 2000/sec, but in error situations, we've
had peaks of up to 20000 packets / sec.

cheers
Chandru

On 16 April 2012 10:31, John-Paul Bader <hukl@REDACTED> wrote:

> Hey Chandru,
>
> how many packets per second did you have to deal with and how big are
> they? Just to have something to compare to.
>
> ~ John
>
> Chandru wrote:
>
>> Hi John,
>>
>> Our RADIUS server which handles our data network is written in Erlang.
>> We've experimented with various values of recbuf and read_packets
>> options for the UDP socket. We also use {active, once}. The receiving
>> process receives a packet and spawns a new process to handle it. That is
>> all it does. The spawned process then executes the rest of the business
>> logic.
>>
>> That won't be your problem. The problem will be to make sure your system
>> is stable while handling all those packets. We use overload control at
>> the receiver. You have to pretty much look at the entire execution path
>> for each packet and ensure there are no bottlenecks. At that kind of
>> load, every little bottleneck shows up sooner or later.
>>
>> cheers
>> Chandru
>>
>> On 15 April 2012 19:08, John-Paul Bader <hukl@REDACTED
>> <mailto:hukl@REDACTED>> wrote:
>>
>>    Dear list,
>>
>>
>>    I'm currently writing a bittorrent tracker in Erlang. While a naive
>>    implementation of the protocol is quite easy, there are some
>>    performance related challanges where I could use some help.
>>
>>    In the first test run as a replacement for a very popular tracker,
>>    my erlang tracker got about 40k requests per second.
>>
>>    My initial approach was to initialize the socket in one process with
>>    {active, once}, handle the message in handle_info with minimal
>>    effort and pass the data asynchronously to a freshly spawned worker
>>    processes which responds to the clients. After spawning the process
>>    I'm setting the socket back to {active, once}.
>>
>>    Now when I switched the erlang tracker live the erlang vm was
>>    topping at 100% CPU load. My guess is that the process handling the
>>    udp packets from the socket could not keep up. Since I'm still quite
>>    new to the world of erlang I'd like to know if there are some best
>>    practices / patterns to handle this massive amount of packets.
>>
>>    For example using the socket in {active, once} might be too slow?
>>    Also the response to the clients needs to come from the same port as
>>    the request was coming in. Is it a problem to use the same socket
>>    for that? Should I pre-spawn a couple of thousand workers and
>>    dispatch the data from the socket to them rather than spawning them
>>    on each packet?
>>
>>    It would be really great if you could give some advice or point me
>>    into the right directions.
>>
>>    ~ John
>>    ______________________________**___________________
>>    erlang-questions mailing list
>>    erlang-questions@REDACTED <mailto:erlang-questions@**erlang.org<erlang-questions@REDACTED>
>> >
>>    http://erlang.org/mailman/__**listinfo/erlang-questions<http://erlang.org/mailman/__listinfo/erlang-questions>
>>    <http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>> >
>>
>>
>>  ______________________________**_________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/**listinfo/erlang-questions<http://erlang.org/mailman/listinfo/erlang-questions>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120416/9785dc56/attachment.htm>