[erlang-questions] Inter-node communication bottleneck

[Jihyun Yu]

I don't think so. pg2 functions are just helper function to
pass pid to other node, and called only once or twice per
sender/receiver process.

To make sure, I made new benchmark program without pg2
functions, and results are same. I tested with N=4 and
Count=1000000. Updated benchmark program is attached.

On Wed, Aug 20, 2014 at 05:16:22PM +0400, Alexander Petrovsky wrote:
> Looks like pg2 may affect you network performance. Could you try get rid of
> it, and make bech again?
> 
> 
> 2014-08-20 15:49 GMT+04:00 Jihyun Yu <yjh0502@REDACTED>:
> 
> > What I want to say is, the benchmark does not scale on multi-core
> > machine while it uses multiple erlang processes to send message.
> >
> > I ran the benchmark with two erlang VM instances, on quad-core
> > laptop. It couldn't saturate all CPU cores, AFAIK because of lock
> > in erlang VM. The benchmark scales when it runs on single erlang
> > VM instance, which shows message passing on single erlang VM
> > scales as expected.
> >
> >
> > On Wed, Aug 20, 2014 at 12:37:02PM +0100, Jon Schneider wrote:
> > > Isn't this simply pushing the performance of Erlang regardless of
> > > networking ?
> > >
> > > I'm sitting at a machine which yes is several years old and running
> > > Windows XP but with a simple ping-pong atom message test between a pair
> > of
> > > processes I get around 125k messages/second.
> > >
> > > Compiling with [hipe] doesn't seem to make any difference.
> > >
> > > Jon
> > >
> > > > Hi, Thanks for reply!
> > > >
> > > > I just tried, but it seems that there is no signigicant difference
> > > > on performance. Here's script which used on benchmarking.
> > > >
> > > > https://gist.github.com/yjh0502/68d6ffce93bcbc425435
> > > >
> > > >
> > > >
> > > > On Wed, Aug 20, 2014 at 02:11:07PM +0400, Alexander Petrovsky wrote:
> > > >> Hello!
> > > >>
> > > >> Have you tried to use:
> > > >>
> > > >> #  erl ... inet_default_connect_options '[{delay_send, true}]'
> > > >>  inet_default_listen_options '[{delay_send, true}]'
> > > >>
> > > >> ?
> > > >>
> > > >>
> > > >> 2014-08-19 16:59 GMT+04:00 Jihyun Yu <yjh0502@REDACTED>:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > There is a prior discusson[1] about inter-node communication
> > > >> bottleneck,
> > > >> > and I experienced same issue on inter-node messaging. I dig into the
> > > >> > issue and it found that there is a lock on inter-node messaging[2]
> > > >> which
> > > >> > causes bottleneck on sending messages to single node. With simple
> > > >> > experiment, I found that there is a limitation on number of messages
> > > >> per
> > > >> > second, not bandwidth. 1Gbps link could be easily satuated by
> > > >> messaging
> > > >> > betweeen two erlang nodes if message size is large enough, e.g. 4KB
> > > >> > binary message. However if message is small, e.g. single integer
> > term,
> > > >> > number of messages per seconds is limited about 200~300k/sec.
> > > >> >
> > > >> > Is there any effort to solve the problem? In fact the problem could
> > be
> > > >> > solved by using external channel like multiple TCP connection pool
> > for
> > > >> > rpc, but I believe using erlang messaging keeps code much simpler.
> > > >> > Actually for me the problem is quite embrassing, because I thought
> > > >> > scalable messaging is a key feature and strength of Erlang.
> > > >> >
> > > >> >
> > > >> > [1]
> > > >>
> > http://erlang.org/pipermail/erlang-questions/2013-December/076232.html
> > > >> > [2]
> > > >> >
> > https://github.com/erlang/otp/blob/OTP-17.1/erts/emulator/beam/dist.c#L1768-L1826
> > > >> > _______________________________________________
> > > >> > erlang-questions mailing list
> > > >> > erlang-questions@REDACTED
> > > >> > http://erlang.org/mailman/listinfo/erlang-questions
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Петровский Александр / Alexander Petrovsky,
> > > >>
> > > >> Skype: askjuise
> > > >> Phone: +7 914 8 820 815
> > > > _______________________________________________
> > > > erlang-questions mailing list
> > > > erlang-questions@REDACTED
> > > > http://erlang.org/mailman/listinfo/erlang-questions
> > > >
> > >
> >
> 
> 
> 
> -- 
> Петровский Александр / Alexander Petrovsky,
> 
> Skype: askjuise
> Phone: +7 914 8 820 815
-------------- next part --------------
-module(test2).

-compile([export_all]).

recv(N, Count) ->
    lists:map(fun(_) ->
        spawn_link(fun() ->
            (fun Recvmsg(0) -> ok;
                 Recvmsg(Remain) ->
                    receive
                        _ -> Recvmsg(Remain-1)
                    end
            end)(Count)
        end)
    end, lists:seq(1, N)).

send(Pids, Count) ->
    lists:map(fun(Pid) ->
        spawn_link(fun() ->
            (fun Sendmsg(0) -> ok;
                 Sendmsg(Remain) ->
                    Pid ! Remain,
                    Sendmsg(Remain - 1)
            end)(Count)
        end)
    end, Pids).

% Spawn N processes, each process sends/receives Count messages
% Sender runs on 'From' node, receiver runs on 'To' node
%
% Following code runs sender on 'foo@REDACTED', receiver on
% 'bar@REDACTED'. Four processes are uses to send/receive messages,
% and each process sends/receives 1,000,000 messages.
%
%  run('foo@REDACTED', 'bar@REDACTED', 4, 1000 * 1000).
run(From, To, N, Count) ->
    Pid = self(),
    spawn_link(To, fun() ->
        Pid ! recv(N, Count)
    end),
    receive
        Pids ->
            spawn_link(From, fun() ->
                send(Pids, Count)
            end)
    end.