[erlang-questions] Idea on stateful server load sharing & fail over

[Kaiduan Xie]

Mihai,

Thanks for the explanation and excellent example. However, you need to
store ALL call information on ALL nodes to the distributed store, this
may not be an optimal solution if the number of nodes and number of
calls are big.

Thanks,

kaiduan

On Wed, Apr 7, 2010 at 11:23 PM, Mihai Balea <mihai@REDACTED> wrote:
>
> On Apr 7, 2010, at 3:09 PM, Kaiduan Xie wrote:
>
>> Thank you Evans for the reply.
>>
>> We are doing process per call instead of process per user because not
>> all users make/receive calls at the same time.
>>
>> We can save all call sate information to shared/distributed database,
>> like Mnesia. There are two issues to be addressed, for example, server
>> A crashes or be removed from the cluster, which server in group will
>> re-construct the gen_gsm/gen_server from the crahsed server A. Saving
>> all call states in all server to distributed database is not optimal
>> also.
>
> You can simply let your dispatcher decide which backend node to redirect calls in case of node crash. When a node receives a message belonging to a call it doesn't know about, it can simply attempt to look it up in the distributed store and recreate the gen_* process. This assumes the dispatcher will forward packets from a call to the same backend node as long as the call is ongoing and the node is up. It also assumes the state you save in the distributed store is sufficient to recreate the internal state of the gen_* process.
>
> Here's an example: let's say call id 42 gets dispatched to node A. Node A updates the distributed call state store whenever needed. At some point node A dies. The dispatcher detects it and decides that call 42 should go to node D from now on, so it starts forwarding call 42 packets to node D. Node starts receiving packets with call id 42, but it has no info about this call. It looks up id 42 in the distributed call state store, finds it, and creates a gen process to handle the call.
>
>>
>> Another problem as I stated, how to dispatch the request to back end
>> servers. Basic hash is not enough because back end server can come up
>> or goes away.
>
> Your dispatcher needs to be aware of what backend nodes are up and running.
> You can monitor nodes in a cluster using net_kernel functions but the default tick time is too long for realtime operations like voip call handling. You can increase the tick frequency, but I don't know how feasible it would be to make it fast enough to work for you. Alternatively, you can implement your own heartbeat protocol with a high enough time resolution.
> Once the dispatcher is made aware of backend nodes coming and going in realtime, then it is a matter of maintaining a table of call ids -> backend nodes and adjusting it on the fly.
>
> Hope this helps
>
> Mihai