[erlang-questions] Idea on stateful server load sharing & fail over

[Nicholas Frechette]

IMO, with a setup like that, you are very likely to bump against scalability
issues. You'll have to save a TON of information, outside the nodes that
will use it thus increasing latency as well.
While this might be fine for thousands or even a few millions, if you plan
on having >5 million simultaneous calls, you'll probably see performance
degrade quite rapidly.

Another alternative would be to run small groups of nodes in mirror:
Nodes A,B,C could handle call X
Nodes D,E,F could handle call Y
Nodes A,B,F could handle call Z
etc
If say node A goes down, you could resume call X on nodes B,C and call Z on
nodes B,F. Because all nodes have identical state, any one of them can pick
up where the others left.
Simply scatter the nodes accross different OS processes, different computers
and you should be fine.
When you initiate a call, simply pick an algorithm that chooses at least 2
physically different computers, and optionally any number of nodes on those
computers to handle it.

I use a similar system at home. One of the nodes handling the request, will
be the master (the one that gets to reply) but all nodes do the work. So
from outside, you simply broadcast the information send to all handling
nodes. If a slave goes down, it doesn't matter, nothing is affected. If the
master goes down, the slaves can elect a new master and it'll resume/reply
with no loss of information.

With the scenario above, you still have a single point of failure where the
process handling the call still has to "post" the information to the
distributed data cache, if it fails before it can post, that call will have
to be dropped. Not so with a scenario like mine above.

Obviously, it'll highly depend on your exact scenario but yeah, your mileage
may vary.

2cents,
Nicholas

On Wed, Apr 7, 2010 at 11:43 PM, Kaiduan Xie <kaiduanx@REDACTED> wrote:

> Mihai,
>
> Thanks for the explanation and excellent example. However, you need to
> store ALL call information on ALL nodes to the distributed store, this
> may not be an optimal solution if the number of nodes and number of
> calls are big.
>
> Thanks,
>
> kaiduan
>
> On Wed, Apr 7, 2010 at 11:23 PM, Mihai Balea <mihai@REDACTED> wrote:
> >
> > On Apr 7, 2010, at 3:09 PM, Kaiduan Xie wrote:
> >
> >> Thank you Evans for the reply.
> >>
> >> We are doing process per call instead of process per user because not
> >> all users make/receive calls at the same time.
> >>
> >> We can save all call sate information to shared/distributed database,
> >> like Mnesia. There are two issues to be addressed, for example, server
> >> A crashes or be removed from the cluster, which server in group will
> >> re-construct the gen_gsm/gen_server from the crahsed server A. Saving
> >> all call states in all server to distributed database is not optimal
> >> also.
> >
> > You can simply let your dispatcher decide which backend node to redirect
> calls in case of node crash. When a node receives a message belonging to a
> call it doesn't know about, it can simply attempt to look it up in the
> distributed store and recreate the gen_* process. This assumes the
> dispatcher will forward packets from a call to the same backend node as long
> as the call is ongoing and the node is up. It also assumes the state you
> save in the distributed store is sufficient to recreate the internal state
> of the gen_* process.
> >
> > Here's an example: let's say call id 42 gets dispatched to node A. Node A
> updates the distributed call state store whenever needed. At some point node
> A dies. The dispatcher detects it and decides that call 42 should go to node
> D from now on, so it starts forwarding call 42 packets to node D. Node
> starts receiving packets with call id 42, but it has no info about this
> call. It looks up id 42 in the distributed call state store, finds it, and
> creates a gen process to handle the call.
> >
> >>
> >> Another problem as I stated, how to dispatch the request to back end
> >> servers. Basic hash is not enough because back end server can come up
> >> or goes away.
> >
> > Your dispatcher needs to be aware of what backend nodes are up and
> running.
> > You can monitor nodes in a cluster using net_kernel functions but the
> default tick time is too long for realtime operations like voip call
> handling. You can increase the tick frequency, but I don't know how feasible
> it would be to make it fast enough to work for you. Alternatively, you can
> implement your own heartbeat protocol with a high enough time resolution.
> > Once the dispatcher is made aware of backend nodes coming and going in
> realtime, then it is a matter of maintaining a table of call ids -> backend
> nodes and adjusting it on the fly.
> >
> > Hope this helps
> >
> > Mihai
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>