[erlang-questions] Strange observation running in an SMP environment.

[Ronny Meeus]

Hello

What I find a really interesting feature about Erlang is that the user
does not need to take care of protecting its data against concurrent
access and all issues that popup when going from the single core to a
multi-core environment.

To play with this I have made a test application which implements a
process ring (like implemented a lot as a starters example).
My code is based on the process ring running on the Raspberry Pi that
makes a LED flash each time the token passes the head of the ring.
The code is available at bitbucket: https://bitbucket.org/meeusr/erl-relay

The machine on which I'm running the test is an Intel i7 (4 cores):
vendor_id       : GenuineIntel
model name      : Intel(R) Core(TM) i7 CPU         860  @ 2.80GHz
cpu MHz         : 2933.000
cache size      : 8192 KB

The basic idea is that I create a LED process that manages the state
of the faked led.
Next to this a ring is created, which contains a head node and a lot
of non-head nodes (1000 in total in the example below).
Once this ring is created, I can send a message to the head process
(hidden behind the relay:start) and this message will be passed around
the ring as indicated in the start.  Each time the messages passes the
head node, the state of the LED is changed by sending a message to the
LED process.
Once all cycles are completed, the time needed to process the messages
is printed.

$  erl
Erlang (BEAM) emulator version 5.6.5 [source] [64-bit] [smp:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5  (abort with ^G)
1> Led = led:start(1).
<0.33.0>
2> Head = relay:create_ring(1000,Led).
<0.1034.0>
3> relay:start(Head,10000).
ok
Finished. Time:3474789

If I force the Erlang VM to run on 1 core only and repeat the same
test as above, I see that the execution time is indeed longer (changes
from 3474789ms to 4186125ms).

$ taskset 1  erl
Erlang (BEAM) emulator version 5.6.5 [source] [64-bit] [smp:4]
[async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.6.5  (abort with ^G)
1>
1> Led = led:start(1).
<0.33.0>
2> Head = relay:create_ring(1000,Led).
<0.1034.0>
3> relay:start(Head,10000).
ok
4>
Finished. Time:4186125
4> q().
ok

I want to stress the system a bit more and create 2 rings, which will
run in parallel:

$ erl
Erlang (BEAM) emulator version 5.6.5 [source] [64-bit] [smp:4]
[async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.6.5  (abort with ^G)
1> Led = led:start(1).
<0.33.0>
2> Head = relay:create_ring(1000,Led).
<0.1034.0>
3> Led1 = led:start(1).
<0.1036.0>
4> Head1 = relay:create_ring(1000,Led1).
<0.2037.0>
5> relay:start(Head,10000),relay:start(Head1,10000).
ok
Finished. Time:7746973
Finished. Time:7824631

An observation is that the execution time is more than doubled
(7746973ms and 7824631ms versus 3474789ms).
But the strange thing I see that if I start to force the VM to run on
1 core, the performance actually becomes better (see below).

Forcing the erlang VM on 1 core ("taskset 1 erl") leads to following result:
7> relay:start(Head,10000),relay:start(Head1,10000).
ok
Finished. Time:6564535
Finished. Time:6755538

Running on 2 cores ("taskset 3 erl") results in:
7> relay:start(Head,10000),relay:start(Head1,10000).
ok
Finished. Time:7186033
Finished. Time:7196646

Running without a taskset ("erl") results in:
7> relay:start(Head,10000),relay:start(Head1,10000).
ok
Finished. Time:7490020
Finished. Time:7814036


So this observation is exactly the opposite of what I would expect.
I would expect that the more cores are being used, the faster the
execution completes but the observation is that the more core are
being used, the slower it becomes.
Now one can say that this is not real parallel work since the token
passes the nodes of the ring one by one, but for the 2 rings example I
would expect that if the number of cores is doubled, the execution
should be finished in more or less the same time as the execution of
the 1 ring instance.

Can somebody explain this?

Many thanks,
Ronny