[erlang-questions] Concurrent processes on multi-core platforms with lots of chatter

[Robert Virding]

I will try and answer these two messages together as the replies are linked.

2009/12/1 Jayson Vantuyl <kagato@REDACTED>

> I'm not too keen on reusing process groups primarily because group leaders
> are used for some really common stuff like IO, which shouldn't affect
> affinity at all.
>

2009/12/1 Håkan Mattsson <hakan@REDACTED>

> On Tue, Dec 1, 2009 at 3:49 AM, Robert Virding <rvirding@REDACTED> wrote:
> > Another solution would be to use the existing process groups as these are
> > not really used very much today. A process group is defined as all the
> > processes which have the same group leader. It is possible to change
> group
> > leader. Maybe the VM could try to migrate processes to the same core as
> > their group leader.
>
> This would not work so well together with the application concept, as
> processes
> in an application has the application master as group leader. Your
> suggestion
> would imply that all processes in an application would be migrated to the
> same
> core.
>

The process group feature was added long-ago as a means of building multiple
more-or-less separate apps (not talking about OTP apps here) each with their
own set of defaults and properties within one erlang node. So each group
could have its own databases, default I/O, current working directories etc,
basically most data that is global to a node today, and not have to share
with other groups. It would be almost like having multiple nodes within a
node. It was (very) loosely modelled on unix process groups. We were
thinking along the Erlang-as-an-OS line here. As the hierarchy-less process
view was already well established, and the Right Thing, we kept the
hierarchy created using process groups to be very loose, it is only one way,
and by default not really noticeable, you inherit your parent's group.

This vision was never really developed and all that process groups are used
for today is default I/O. This also means that they are very sparingly used.

So my direct comments:

- I think the fact that process groups today are used for stuff common to
the group is something which should affect affinity. Where better to put
stuff common to a group of processes than close to the processes?

- As process groups are used for very little, then they are very little
used. There is nothing as far as I know which demands that an application is
in only one group, it is done so today because that is sufficient for
process groups provide. In the same way you have supervisor trees there is
nothing which prohibits you from having group leader trees with different
types of group leaders depending on where the group common stuff is to be
handled; a group leader could pass on its requests up the tree until the
right group leader is reached.

I don't like the idea of adding new features to the language if there is
something already there which could be used, the language is growing enough
as it is, not all bad of course.

Robert

P.S. There is a bug in my argument which is correctable. What it is is left
as problem for the reader. :-)