[erlang-questions] Inter-node communication bottleneck

[Jihyun Yu]

What I want to say is, the benchmark does not scale on multi-core
machine while it uses multiple erlang processes to send message.

I ran the benchmark with two erlang VM instances, on quad-core
laptop. It couldn't saturate all CPU cores, AFAIK because of lock
in erlang VM. The benchmark scales when it runs on single erlang
VM instance, which shows message passing on single erlang VM
scales as expected.


On Wed, Aug 20, 2014 at 12:37:02PM +0100, Jon Schneider wrote:
> Isn't this simply pushing the performance of Erlang regardless of
> networking ?
> 
> I'm sitting at a machine which yes is several years old and running
> Windows XP but with a simple ping-pong atom message test between a pair of
> processes I get around 125k messages/second.
> 
> Compiling with [hipe] doesn't seem to make any difference.
> 
> Jon
> 
> > Hi, Thanks for reply!
> >
> > I just tried, but it seems that there is no signigicant difference
> > on performance. Here's script which used on benchmarking.
> >
> > https://gist.github.com/yjh0502/68d6ffce93bcbc425435
> >
> >
> >
> > On Wed, Aug 20, 2014 at 02:11:07PM +0400, Alexander Petrovsky wrote:
> >> Hello!
> >>
> >> Have you tried to use:
> >>
> >> #  erl ... inet_default_connect_options '[{delay_send, true}]'
> >>  inet_default_listen_options '[{delay_send, true}]'
> >>
> >> ?
> >>
> >>
> >> 2014-08-19 16:59 GMT+04:00 Jihyun Yu <yjh0502@REDACTED>:
> >>
> >> > Hi,
> >> >
> >> > There is a prior discusson[1] about inter-node communication
> >> bottleneck,
> >> > and I experienced same issue on inter-node messaging. I dig into the
> >> > issue and it found that there is a lock on inter-node messaging[2]
> >> which
> >> > causes bottleneck on sending messages to single node. With simple
> >> > experiment, I found that there is a limitation on number of messages
> >> per
> >> > second, not bandwidth. 1Gbps link could be easily satuated by
> >> messaging
> >> > betweeen two erlang nodes if message size is large enough, e.g. 4KB
> >> > binary message. However if message is small, e.g. single integer term,
> >> > number of messages per seconds is limited about 200~300k/sec.
> >> >
> >> > Is there any effort to solve the problem? In fact the problem could be
> >> > solved by using external channel like multiple TCP connection pool for
> >> > rpc, but I believe using erlang messaging keeps code much simpler.
> >> > Actually for me the problem is quite embrassing, because I thought
> >> > scalable messaging is a key feature and strength of Erlang.
> >> >
> >> >
> >> > [1]
> >> http://erlang.org/pipermail/erlang-questions/2013-December/076232.html
> >> > [2]
> >> > https://github.com/erlang/otp/blob/OTP-17.1/erts/emulator/beam/dist.c#L1768-L1826
> >> > _______________________________________________
> >> > erlang-questions mailing list
> >> > erlang-questions@REDACTED
> >> > http://erlang.org/mailman/listinfo/erlang-questions
> >> >
> >>
> >>
> >>
> >> --
> >> Петровский Александр / Alexander Petrovsky,
> >>
> >> Skype: askjuise
> >> Phone: +7 914 8 820 815
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
> >
>