distributed performance test

[Sean Hinde]

Hi,

We have seen such problems when networks/NICs are set up incorrectly 
(e.g. one end forced to 10/half duplex, the other to 100/full duplex).

I suspect you will spend a very long time searching for huge delays in 
the message passing inside Erlang :)

Sean

On 25 Apr 2005, at 01:17, Serge Aleynikov wrote:

> Thanks Matthias,
>
> I'd like to also confirm that I am not getting this weird behavior on 
> other computers, yet this particular machine (devlinuxpro2) is quite 
> powerful: 4CPUs, 4G RAM, 100M Eithernet, 100% idle, and I am not 
> certain on how to interpret this result.
>
> This time I am showing results of three tests:
> 1. A client and a server are on different nodes running on the same 
> host (Linux).
>
> 2. A client and a server are on different nodes running on different 
> hosts (both have Linux).
>
> 3. A client and a server are on different nodes running on different 
> hosts with less powerful hardware (a Windows PC and a Linux server)
>
> Test# CLIENT          SERVER          #ofClients Calls/sec Ping(us)
> ===== ======          ======          ========== ========= ========
> 1a    n1@REDACTED n2@REDACTED 1          5714
> 1b    n1@REDACTED n2@REDACTED 5	         3067
>
> 2a    n1@REDACTED  n2@REDACTED 1          1727      150us/0% 
> loss
> 2b    n1@REDACTED  n2@REDACTED 5	         54        <---- !!!!
>
> 2c    n3@REDACTED n1@REDACTED  1          4115      150us/0% 
> loss
> 2d    n3@REDACTED n1@REDACTED  5	         50        <---- !!!!
>
> 3a    a@REDACTED     n1@REDACTED  1          842       400us/0% 
> loss
> 3b    a@REDACTED     n1@REDACTED  5	         603
>
> and just for the sake of simmetry:
>
> 2e    n2@REDACTED  a@REDACTED     1          582      400us/0% 
> loss
> 2f    n2@REDACTED  a@REDACTED     5	  16       <---- !!!!
>
> So, what are the reasonable questions to ask at this point about 
> devlinuxpro2?
>
> Since two nodes running on the same host don't express this problem, 
> it must be due to a mulfinctioning network interface.  On the other 
> hand, why in that case, I don't see any problem when running a single 
> client?  So, assuming that something _is_ wrong with the network 
> interface, how can such a problem be troubleshot (tcpdump dosn't show 
> anything suspicious either)?
>
> Ideally, I'd like to be able to see some stats from the message 
> passing layer in Erlang in order to determine the speed of packets of 
> interest arriving to the node, but before they are dispatched to the 
> process' mailbox.  This way, I would know that the delay is in fact 
> external to Erlang.  Is there a way to get that?
>
> Thanks.
>
> Serge
>
>
> Matthias Lang wrote:
>
>> Ignore my previous post... The source was included in another message.
>> I re-ran your tests on two machines and I do _not_ see the unexpected
>> behaviour you're seeing. You reported:
>>  > CLIENT          SERVER          THREADS PERFORMANCE (Calls/sec)
>>  > ======          ======          ======= =======================
>>  > n2@REDACTED	n2@REDACTED	1	238095
>>  > n2@REDACTED	n2@REDACTED	5	43478
>>  >  > n1@REDACTED	n2@REDACTED 1	2364
>>  > n1@REDACTED	n2@REDACTED 5	71        !!!  Why?
>> Repeating this on a PC and a laptop:
>> CLIENT          SERVER          THREADS PERFORMANCE (Calls/sec)
>> ======          ======          ======= =======================
>> a@REDACTED      a@REDACTED      1       257997
>> a@REDACTED      a@REDACTED      5        52714
>> b@REDACTED     a@REDACTED      1         1193
>> b@REDACTED     a@REDACTED      5          319
>> a@REDACTED      b@REDACTED     1          905
>> a@REDACTED      b@REDACTED     5          340
>> The machines were "mostly idle" when I ran the tests. I used R9C on
>> both. The network between them is slow (10Mbit, hub). Ping roundtrip
>> time is about 400us.
>> Matthias
> -module(test_client).
>
> % API
>
> -export([test/3]).
>
> % Internal
>
> -export([do_test/3]).
>
> test(Node, Parallelizm, Count) when is_integer(Count), Count > 0 ->
>     lists:foreach(
>         fun(I) ->
>             erlang:spawn(?MODULE, do_test, [Node, I, Count])
>         end,
>         lists:seq(1, Parallelizm)).
>
> do_test(Node, Instance, Count) ->
>     StartTime = get_tick_count(),
>     {NSuccess, NFail} = do_test_repeat(Node, Count, {0, 0}),
>     Time = (get_tick_count() - StartTime)/1000,
>     Speed = if Time =/= 0.0 -> Count / Time; true -> 0.0 end,
>     io:format("Client ~2w done. AvgTime=~.3f (~.3f c/s), Failed=~p, 
> Success=~p~n",
>                 [Instance, Time / Count, Speed, NFail, NSuccess]).
>
> do_test_repeat(_Node, 0, Stats) ->
>     Stats;
> do_test_repeat(Node, I, {S, F}) ->
>     Stats = case test_server:ping(Node) of
>             ok -> {S+1, F};
>             _  -> {S, F+1}
>             end,
>     do_test_repeat(Node, I-1, Stats).
>
> get_tick_count() ->
>     {A,B,C} = now(),
>     (A-1051) * 1000000000 + B * 1000 + C div 1000.
>
> -module(test_server).
> -behaviour(gen_server).
>
> %% External exports
> -export([start/0, ping/0, ping/1]).
>
> %% gen_server callbacks
> -export([init/1, handle_call/3, handle_cast/2,
>          handle_info/2, terminate/2, code_change/3]).
>
> %% API
>
> start() ->
>     gen_server:start({local, ?MODULE}, ?MODULE, [], []).
>
> ping() ->
>     ping(node()).
>
> ping(Node) ->
>     gen_server:call({?MODULE, Node}, ping, 10000).
>
> %---------------------------------------------------------------------
> % Callback functions
> %---------------------------------------------------------------------
>
> init([]) ->
>     {ok, none}.
>
> handle_call(ping, _From, State) ->
>     {reply, ok, State}.
>
> handle_cast(Msg, State) ->
>     {stop, {unknown_cast, Msg}, State}.
>
> handle_info(_Info, State) ->
>     {noreply, State}.
>
> code_change(_OldVsn, State, _Extra) ->
>     {ok, State}.
>
> terminate(_Reason, _State) ->
>     ok.
>