[erlang-questions] Inter nodes messaging bottleneck

[Eli Cohen]

Hello,

I'm facing a bottleneck issue between two nodes.
I have several nodes in my cluster, each on different machine.
Physical interface between the machines is 10G.
On one node I have about 100 processes receiving packets from UDP sockets
(~200 sockets) and banging Erlang messages towards a registered process
located on remote node (also remote machine).
In the reduced scenario the receiving process is a gen_server which does
nothing with the messages - just counts the received messages (internal
state counter, not a DB).
By checking this internal counter I can say that maximum rx rate of this
process stands on ~150K messages per second and there is no messages
waiting in its queue.
Increasing the transmitting node feed does not affect this number and
finally the transmitting processes queues start to grow.
Just to clarify that these 150K messages are about 120Mbps (over 10G line).
Actually I expected to see about 1M.

When I'm running these both nodes on same machine - the maximum received
messages rate grows to ~400K per second.

I've tried to set port parallelism (+spp) flag, increase distribution
buffer busy limit (+zdbbl), set kernel poll (+K) - no significant
difference.

Currently my conclusion is that the bottleneck is somewhere on transmit
between nodes.

Is there any flags/parameters that may affect this area?

Any ideas, comments, help are more than welcomed :-)
Thanks,
Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20131210/a331ae53/attachment.htm>