[erlang-questions] UDP client/server performance

[Ronny Meeus]

Hello

my final goal is to implement a statsd like application completely in
Erlang (as a exercise to learn more about the various aspects of
Erlang). The first step for me is to implement a UDP echo server to
have a feeling about the performance.
Since this was very low in my opinion (the server handled only 5000
messages per second), I wrote the same in C. While running the etop I
also observed that the system was using a lot of memory while sending
the messages.

Basically both applications do the same thing: the client is sending a
string "TEST" and the server echos it back.

The bad performance of the Erlang solution triggered me to send a mail
to the community, maybe I was doing something wrong.
So it might be good to have a look at the code I have created and
check whether it is using the gen_udp in a correct and most performant
way.

I agree that it is better to improve the gen_udp code so that it is
beneficial for all applications, but I'm afraid that my skills are not
good enough for that.

Ronny

On Wed, Aug 15, 2012 at 12:22 PM, Kenneth Lundin
<kenneth.lundin@REDACTED> wrote:
> Hi Ronny,
>
> I think you are a little bit fast in drawing the conclusion that you
> must look at other solutions
> than just using the standard gen_udp module.
>
> If you have found some real bottle neck in performance we should of
> course try to find out why the difference between C and Erlang is so
> big, I don't think it should be that big.
>
> In case there is something unnecessarily inefficient going on when
> using gen_udp we should
> try to address that instead of looking at more complex and not so
> general alternatives
>
> A deeper analysis of what your test is doing would be good, is it
> comparable with the C case?
> Can the difference be explained etc.
>
> I am not convinced that you will get better speed the alternative ways
> you have been recommended. The socket code is written in C even when
> you use gen_udp so I don't
> really see the big difference. I also suppose you in reality want to
> do something with the
> data on the Erlang side and that would probably change the
> characteristics. Another thing is if there is multiple cores involved
> etc.
>
> Kenneth , Erlang/OTP Ericsson
>
> On 8/14/12, Ronny Meeus <ronny.meeus@REDACTED> wrote:
>> Is there any code available that implements a NIF to have access to UDP
>> sockets?
>> I have found the procket implementation but it looks like it is using
>> a separate process to communicate with the socket. This sounds like
>> overkill to me for this test.
>>
>> Ronny
>>
>> On Mon, Aug 13, 2012 at 10:29 PM, Ronny Meeus <ronny.meeus@REDACTED>
>> wrote:
>>> Hello
>>>
>>> I did a quick check.
>>> - The values between 2 machines are comparable for Erlang.
>>> - For the C code there is a difference: the async version drops from
>>> 700Kpks/sec to 400Kpks/sec while the sync version drops from
>>> 86Kpkts/sec to 10Kpkts/sec.
>>>
>>>
>>> On Mon, Aug 13, 2012 at 9:42 PM, JD Bothma <jbothma@REDACTED> wrote:
>>>> I'm curious:
>>>>
>>>> What are the numbers like (c versus erlang) when it's between two
>>>> physical machines? Or between two (vmware or virtualbox, not xen)
>>>> virtual machines?
>>>>
>>>> Or said otherwise: does the fact that it's the same machine, network
>>>> stack and loopback interface affect the comparison between c and
>>>> erlang?
>>>>
>>>> JD
>>>>
>>>> On 13 August 2012 21:11, Ronny Meeus <ronny.meeus@REDACTED> wrote:
>>>>> Hello
>>>>>
>>>>> I have created a simple UDP based client/server application.
>>>>> The code is available on bitbucket
>>>>> (https://bitbucket.org/meeusr/erl-examples) in the networking
>>>>> directory.
>>>>>
>>>>> The server is just an echo server running in 1 process.
>>>>> The client can behave like a sync client (sending 1 message and
>>>>> waiting for the reply) or like an async client (sending X messages
>>>>> before waiting for all the replies).
>>>>> I see that my PC (Intel(R) Quad code i7 CPU 860  @ 2.80GHz) is able to
>>>>> process something like 7K - 10K messages per second. Both the sync and
>>>>> the async are in the same order of magnitude. Please note that I use
>>>>> the loopback interface.
>>>>>
>>>>> $ erl +sbt db
>>>>> Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:4:4]
>>>>> [async-threads:0] [hipe] [kernel-poll:false]
>>>>>
>>>>> Eshell V5.9.1  (abort with ^G)
>>>>> 1> c(gen_udp_test).
>>>>> gen_udp_test.erl:47: Warning: variable 'Data' is unused
>>>>> gen_udp_test.erl:47: Warning: variable 'Msg' is unused
>>>>> gen_udp_test.erl:62: Warning: variable 'Data' is unused
>>>>> gen_udp_test.erl:62: Warning: variable 'Msg' is unused
>>>>> {ok,gen_udp_test}
>>>>> 2> gen_udp_test:start_server().
>>>>> true
>>>>> ...
>>>>> 12> gen_udp_test:client_async(5000).
>>>>> stop : 841482
>>>>> ok
>>>>> 13> gen_udp_test:client_async(6000).
>>>>> stop : 1082679
>>>>> ok
>>>>>
>>>>> The timings are the number of usec for the processing of the X number
>>>>> of messages (round-trip).
>>>>>
>>>>> If I implement the same application in C (client and server are 2
>>>>> applications), I see that the application performs an order of
>>>>> magnitude better for the sync solution (86Kpkts/sec) and 100 times
>>>>> better (700Kpkts/sec) for the async version.
>>>>> The code for this is also available in the repo on bitbucket.
>>>>>
>>>>> I know there this test is only looking to a part of the application
>>>>> (transport of the data), but I find the difference rather big.
>>>>> Is this normal or is there something wrong in my Erlang code?
>>>>>
>>>>> Thanks.
>>>>> Regards,
>>>>> Ronny
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> erlang-questions@REDACTED
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>