[erlang-questions] gen_tcp:send() very expensive

[Edwin Fine]

It would help if you posted the hardware and software systems and versions
(CPU, memory, OS, Erlang) you are using. Are you running everything on one
machine? What speed of LAN are you using?

In terms of the architecture, are the TCP/IP connections persistent between
client and the system sending the messages, or is there a connect for each
message (obvious, I know, but still).

Is there a message-level protocol? A 2-byte message does not seem long
enough to involved a protocol, but perhaps I misunderstand.

In an application I am working on, I used Apache JMeter (Java app) running
50 clients on a Windows computer, and managed to send an aggregate of around
675 500-byte XML messages per second (and get an XML response to each
message) to a system running Erlang R12B-3 (Linux 64-bit), and the CPU usage
on the Linux system was about 25%. The Erlang application's CPU usage was
about 60 - 75% within the VM (using appmon).

On thing I had to be sure of is that I was using the Erlang run-time flag
'+K true' to use kernel poll on Linux. I also had to tweak some Windows
registry values to get better TCP/IP performance. If you are using Erlang on
Windows then I all I can say is that I am not sure that's the best platform
for it. Finally, I found that the way I sent messages from Erlang made a
huge difference to performance and CPU usage. Use a TCP/IP sniffer tool
(e.g. Wireshark) to see the size of the frames that come from your Erlang
system. If they are small, then you are probably somehow preventing Erlang
from buffering up the different fragments of data to reduce the number of
actual TCP/IP transmissions done. That happened to me, and I saw dismal
performance. What I did to fix that was to ensure that every message to be
sent on TCP/IP was spawned into its own Erlang process, passing in an open
socket. (Remember that process creation in Erlang is extremely fast and has
very low overhead - of the order of sending an Erlang message). The sniffer
then showed that instead of sending a zillion tiny frames, it was now
sending much fewer, larger ones. For example, many frames were 56 bytes
long. After making the send asynchronous, the frames were close to 1400
bytes. I think the improvement resulted because I was feeding the socket in
a sort of sequential way by waiting for the gen_tcp:send() to return ok,
then fetching the next message to send. I believe that doing this prevented
Erlang's networking layer from buffering the send data. When I spawned a
process to send, I was no longer waiting for the ok before sending the next
message to the same socket.

Someone else who had this issue actually buffered them up himself, but I
think my approach also works. Maybe spawning the sends is akin to buffering
them up. Either way, the principle is the same.

Other than that, there are many TCP/IP parameters that can be modified both
on Windows and in Erlang, and these can make a big difference to the
performance depending on the situation.

Hope this helps.

On Sat, Jul 12, 2008 at 12:59 PM, Mark Geib <mark.geib@REDACTED> wrote:

> I am new to erlang and writing a program to replace an existing Java
> program that accepts multiple network tcp connections, upto about 400,
> and then also excepts 'receive' connections from clients, again in the
> hundrends.. The Java code then re-distributes all the messages from all
> the 'source' connections to all the connected clients.
>
> The Java program is running at about 35% of the machines cpu. When I
> tried my first cut of replacing this with erlang I was surprised to find
> that the erlang program rapidly consumed the machine and quickly stopped
> functioning...
>
> SO, I have been investigating and find that the erlang message passing
> is VERY efficient, but the actual sending of the message to all the
> connected clients is a HUGE cpu consumer. In my test case I am
> generating about 2000 message / second and without sending the tcp
> messages uses about 10% of the test machine. With one client connecting
> the cpu usage increases to nearly 100%.
>
> Is this expected, or is there an option I need to set improve efficiency
> of the network writing. The message sizes vary from 2 - 2000 bytes.
>
> Sorry for the long post.
>
> Mark.
> --
> Principal Engineer
> Cheyenne Software Engineering
> mark.geib@REDACTED / 35-215
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>


-- 
The great enemy of the truth is very often not the lie -- deliberate,
contrived and dishonest, but the myth, persistent, persuasive, and
unrealistic. Belief in myths allows the comfort of opinion without the
discomfort of thought.
John F. Kennedy 35th president of US 1961-1963 (1917 - 1963)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080712/10e858f1/attachment.htm>