[erlang-questions] Concurrent processes on multi-core platforms with lots of chatter

[Wallentin Dahlberg]

2009/11/30 Evans, Matthew <mevans@REDACTED>

> This is one reason why I thought of something along the lines of a process
> group (Zoltan’s idea has merits too). By that I mean the developer knows
> that process A, B, C and D are going to be involved in a large amount of IPC
> for the next few seconds (or longer, or shorter). The developer shouldn’t
> care if it is single core or multi-core. All they are saying to the runtime
> is “please ensure that these processes are on the same scheduler for the
> next X micro seconds since they are going to be communicating a lot”. Now
> the runtime could ignore that request if it thought things could run faster
> if they were on different cores.
>

Yes, the runtime system can only make decisions by observing the past, it
knows nothing about the future. When sudden shifts in processes messaging
patterns occurs, or completly new processes with new patterns emerges, the
scheduler won't know anything about how to handle it and probably won't
schedule those processes optimal. To do it reasonable we have to assume that
messaging patterns converge to some steadystate in the system. We have to do
some thinking on this to see if it is a reasonable assumption. The point is
the system should be able to learn the best scheduling strategy given enough
time.

>
>
> I think my point is that you are correct in that developers should not care
> what the CPU topology is, but at the same time you don’t want the VM making
> bad choices too.
>
We could help the scheduler along by letting the user define some
abstractions. Something that scheduler can look at if it feels uncertain,
like process groups. Or perhaps it can take advantage of some of the design
patterns and behaviours already in OTP?

It will be alot of tweaking for finding the right balance between good
enough scheduling, information, decision calculation and overall cost. But
messaging patterns are only one part of the problem.

When I think about this scheduling problem I often think about it like
Zoltan describes it, a force graph. It is just dead simple that way.

A second order diff.eq. like nature would have designed it. The forces
involved to each scheduler are just rules that has to obeyed, each term is a
criteria, message passing patterns, cpu topology and the cost of moving,
memory hierachy, priorities, load differences, user defined hints, etc and
then just do the math each scheduling iteration. Now, if we just could make
accounting and statistics collecting cost free.

I have to revisit this with Rickard.

Regards,
Björn-Egil


 ------------------------------
>
> *From:* Wallentin Dahlberg [mailto:wallentin.dahlberg@REDACTED]
> *Sent:* Monday, November 30, 2009 2:46 PM
> *To:* Evans, Matthew
> *Cc:* erlang-questions@REDACTED
> *Subject:* Re: [erlang-questions] Concurrent processes on multi-core
> platforms with lots of chatter
>
>
>
> 2009/11/30 Evans, Matthew <mevans@REDACTED>
>
> First the good news: When running tests that do more than just message
> passing the SMP features of R13B02 are leaps and bounds over R12B05 that I
> was running previously. What I have however noticed is that in a pure
> messaging test (lots of messages, in a tight loop) we appear to run into
> caching issues where messages are sent between processes that happen to be
> scheduled on different cores. This got me into thinking about a future
> enhancement to the Erlang VM: Process affinity.
>
> In this mode two or more processes that have a lot of IPC chatter would be
> associated into a group and executed on the same core. If the scheduler
> needed to move one process to another core - they would all be relocated.
>
> Although this grouping of processes could be done automatically by the VM I
> believe the decision making overhead would be too great, and it would likely
> make some poor choices as to what processes should be grouped together.
> Rather I would leave it to the developer to make these decisions, perhaps
> with a library similar to pg2.
>
>
> Process/scheduler affinity has been discussed at length before and there
> are several schools of thought on the matter.
>
> It is true that accounting and statistics gathering in the runtime has some
> overhead. However, this should be handled by the scheduler. The user (the
> erlang developer) might not have all the information at hand and if he
> doesn't he has to collect them which also costs. It would be easier for the
> system to schedule the process right. The scheduler still has collect other
> information and make decisions on memory models, memory distance, number of
> processing units, process to process message affinity, load balancing and
> other characteristics. It would be reasonable easy to model a scheduler
> algorithm after these charateristics and we can make it much more dynamic in
> the runtime system.
>
> Ideally the user shouldn't be concerned about the hardware he is running
> on.   Thats my take on the situation anyway.
>
> Regards,
> Björn-Egil
>