[erlang-questions] gproc : 2 nodes out of sync behaviour.

[Ulf Wiger]

On 1 Jul 2012, at 17:01, Morgan Segalis wrote:

> Gproc may make life more interesting, but right now, I certainly know that gproc made my life easier, thanks to you :-)

Why, thanks. :)


> When you say : " since gen_leader didn't use to have a way to handle netsplits."
> 
> 1- this means that gen_leader handles netsplits now ?

Yes, later versions of gen_leader, e.g. Garrett Smith's fork, [1], have some point in the code where they detect a netsplit and call a new election.

[1] https://github.com/garret-smith/gen_leader_revival/blob/master/src/gen_leader.erl#L874
I picked Garrett's purely on the basis of the Github network graph, seeing as it is farthest to the right, and has recent netsplit-related commits. It's not a value judgement based on having tried them or deeply studied the code.

> 2- If so, gproc_dist would only need a way to know when a netsplits happened, right ?

Well, a few things may need to be addressed:

1. Currently, gen_leader simply calls a new election and decides who's the new leader. This *may* be sufficient - probably is in many cases - but in some cases, what's happened is that one node has become isolated, and while gen_leader may think it has the best claim to the throne, the people may disagree. The choice that might cause the least disruption may be if the application reads the situation and picks the node that is actually busy servicing customers. A possibility could then be that gen_leader first asks the applications: "these leaders are in contention; do you have a favorite?" Answering this question with anything other than 'no' can risk destabilizing the system unless the two candidates disagree. Gen_leader *might* be able to detect t his, and call an election instead.

2. Next problem is the reconciliation. There are essentially two ways:
   a) Once a new leader is elected, you call elected() with some info signaling that there has been a
       netsplit, and rely on it to take care of things. It might not be possible to reuse elected() (not in a 
      position  to check right now).
   b) Enter a protected state machine, similar to the election state machine, and allow the leaders to 
        signal each other asynchronously until they agree.

The two may be able to combine, by allowing the elected() callback order a reconciliation procedure.

I'm sure the hackers at e.g. Basho could offer good insights into this.

BR,
Ulf W

> 3 - if 1 & 2 are not true then would it be possible, in your opinion, to stop & start gproc and re-register every value so every cluster are in sync again ?
> 
> Le 1 juil. 2012 à 13:49, Ulf Wiger a écrit :
> 
>> It's a feature of gproc, carefully crafted to make life more interesting. ;-)
>> 
>> There is no resynch after netsplit in gproc, since gen_leader didn't use to have a way to handle netsplits. Still, there is no hook to inform the callback (gproc_dist) about what's happened.
>> 
>> One way to deal with this, is to set -kernel dist_auto_connect false, and add a "backdoor ping" (e.g. over UDP). If you get a ping from a known node that's not in the nodes() list, you have a netsplit situation. You can then select which node(s) to restart. After restart, normal synch will ensue, and since the nodes never auto-connected, you will have concistency (but quite possibly data loss, of course).
>> 
>> BR,
>> Ulf W
>> 
>> Ulf Wiger, Feuerlabs, Inc.
>> http://www.feuerlabs.com
>> 
>> 1 jul 2012 kl. 13:36 skrev Morgan Segalis <msegalis@REDACTED>:
>> 
>>> Hello everyone,
>>> 
>>> I have 2 nodes which use gproc.
>>> Both are well connected to each other…
>>> But sometimes (doesn't happen really often, but it does) both server gets disconnected to each other, once their are connected again, gproc is out of sync.
>>> 
>>> Here's what happen : 
>>> 1- A is connected to B.
>>> 2- a new value X set by A is saw by B
>>> 3- a new value Y set by B is saw by A
>>> -------- they get disconnect for a second or two --------
>>> 4- Clusters lost connection
>>> -------- they reconnect ----------
>>> 5- Clusters regain connection
>>> 6- the old value X set by A is not saw anymore by B
>>> 7- the old value Y set by B is not saw anymore by B
>>> 8- a new value Z set by A is saw by B
>>> 9- a new value V set by B is not saw by A
>>> 
>>> how come in "8" the new value Z set by A is saw by B and in "9" a new value V set by B is not saw by A ?
>>> I know that there is a leader, which is probably B, but I can't explain why new value are not seen symmetrically.
>>> what should I do for reconnecting correctly both cluster, so old value and new value are saw in both cluster again ?
>>> 
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
> 

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com