[erlang-questions] Inter nodes messaging bottleneck

[Valentin Micic]

Hi Eli,

If you're sending more then you're receiving whilst using UDP as a transport, best bet would be to increase tx and rx buffers on the UDP socket… but then again, I am confused as to why are you using 200 UDP sockets -- care to share your reason for that?

Kind regards

V/


On 10 Dec 2013, at 5:09 PM, Eli Cohen wrote:

> Hello,
> 
> I'm facing a bottleneck issue between two nodes.
> I have several nodes in my cluster, each on different machine.
> Physical interface between the machines is 10G.
> On one node I have about 100 processes receiving packets from UDP sockets (~200 sockets) and banging Erlang messages towards a registered process located on remote node (also remote machine).
> In the reduced scenario the receiving process is a gen_server which does nothing with the messages - just counts the received messages (internal state counter, not a DB).
> By checking this internal counter I can say that maximum rx rate of this process stands on ~150K messages per second and there is no messages waiting in its queue. 
> Increasing the transmitting node feed does not affect this number and finally the transmitting processes queues start to grow. 
> Just to clarify that these 150K messages are about 120Mbps (over 10G line). Actually I expected to see about 1M.
> 
> When I'm running these both nodes on same machine - the maximum received messages rate grows to ~400K per second.
> 
> I've tried to set port parallelism (+spp) flag, increase distribution buffer busy limit (+zdbbl), set kernel poll (+K) - no significant difference. 
> 
> Currently my conclusion is that the bottleneck is somewhere on transmit between nodes.
> 
> Is there any flags/parameters that may affect this area?
> 
> Any ideas, comments, help are more than welcomed :-)
> Thanks,
> Eli
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions