[erlang-questions] gen_server clustering strategy

[Scott Lystig Fritchie]

Following-up many days after the last posting, sorry.

Knut Ivar Nesheim <knutin@REDACTED> wrote:

kin> * We need to be able to send messages from every process to any
kin> other process, so we could use some form of registry. Using the
kin> global module would get us going, but not take us all the way
kin> (which is something we can worry about later).

'global' will allow prototype code to work in a small development
system, but it won't scale to what you need.  Previous messages have
pointed to reasons why.

kin> * We have considered writing our own process registry, based on ets
kin> or mnesia or redis or whatever [...]

Mnesia and Redis and Riak would work quite well for that, ...

kin> * We would like to add nodes without disturbing the processes
kin> already running, so consistent hashing must be used if hashing...

... though it isn't clear to me (haven't followed recent Redis
developments) if Redis can cope with changing cluster sizes dynamically.
Mnesia and Riak both can.

kin> Our worry with the cluster of identical nodes, the "mesh" described
kin> earlier, is that there will be too many messages passing around.
kin> [...]
kin> Our worry with the cluster that includes "proxy" nodes, is that
kin> they must maintain the state of the cluster in order to route
kin> accordingly.  At least they must know where to send the requests
kin> using some form of hashing.

Er, you could build a system that routes messages like that, but neither
Mnesia nor Riak use that method, so one need not worry about it.

kin> To this end, building on top of riak_core might be a very good
kin> option. However, we feel that riak_core is pretty complex to
kin> understand fully and would prefer a simpler solution.

Riak Core's callback functions aren't terribly complex to implement.
Though for your name->pid process registry, using the as-is Riak KV
application would be sufficient: KV is already using Core to figure out
where your key to map name->pid would be.  

kin> If riak is running on 60 nodes, then the question for us is how
kin> much message passing is going on.

The hash calculation is O(1) with respect to CPU time, and fetching the
value is O(1) with respect to number of messages required.

kin> Also, what kind of overhead can we expect if we monitor processes
kin> on other nodes? Our understanding is that the information regarding
kin> pids that die, will be sent to any node in the cluster anyway, so
kin> actually monitoring and receiving the info won't make that much of
kin> a difference as it is already there.

Er, if I'm understanding your understanding, no.  Information on a
monitored process P is only sent to interested parties, i.e. those procs
that actually created a monitor relationship with P.

One thing that you didn't mention is what happens to the application
when one of your gen_server worker processes dies.  Do those processes
contain state that must be persistent?  If that state must be
persistent, then you have another data management task that you haven't
asked about yet.

-Scott