Hi Ulf,<div><br></div><div>I just encountered some unexpected behaviour in application failover. When a distributed application stops because its top-level supervisor exits with a <font class="Apple-style-span" face="'courier new', monospace">reached_max_restart_intensity</font> error, it will not be restarted on the other node. This, I guess, is consistent with the documentation, which says that the application will be restarted only if the node currently running the application goes down, not if the application stops with an error.</div>
<div><br></div><div>However, if the application is installed as a permanent application, the whole Erlang node will be terminated as soon as the application stops. But even in this case, the application will not be restarted on the other node. In fact, calling a<font class="Apple-style-span" face="'courier new', monospace">pplication:info()</font> on the other node claims that the application is still running as <font class="Apple-style-span" face="'courier new', monospace">{ distributed, OriginalNode }</font>, even though <font class="Apple-style-span" face="'courier new', monospace">OriginalNode </font>no longer exists. </div>
<div><br></div><div>Is this the intended behaviour? I had hoped that, by installing an application as permanent, I could make it failover to a standby node in case of an unrecoverable software-error *), but apparently that doesn't work.</div>
<div><br></div><div><br></div><div>*) Of course, failing over in case of a pure software error does not make a lot of sense since the application would crash on the other node as well. But quite a few errors are causes by unexpected changes in the environment that are handled badly by the software, and those are the ones I'm looking at.</div>
<div><br><br><div class="gmail_quote">On Thu, Mar 31, 2011 at 9:15 PM, Ulf Wiger <span dir="ltr"><<a href="mailto:ulf.wiger@erlang-solutions.com">ulf.wiger@erlang-solutions.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div style="word-wrap:break-word"><div><br></div><div>Hi Jeroen,</div><div><br></div><div>What we used to do was to use start phases and global:re_register_name/2</div><div><br></div><div>Application start phases can be used to do e.g. global name registration in an ordered fashion.</div>
<div><br></div><div>We had, for historical reasons, the following start phases at AXD 301:</div><div><br></div><div>- start: processes started, basic initialisation</div><div>- takeover: ignored unless application start type was {takeover, FromNode}</div>
<div>- go: to register global names, do other forms of complex initialisation, etc.</div><div><br></div><div>In the takeover phase, the takeover callback would typically do gen_server calls to the application processes. These could in their turn do a gen_server call to their counterpart on the other side, fetching state, telling the other side to start forwarding requests, etc. A good order of doing things would be to first re_register the global name, then call the other process to fetch state.</div>
<div><br></div><div>See:</div><div><a href="http://www.erlang.org/doc/man/app.html" target="_blank">http://www.erlang.org/doc/man/app.html</a></div><div><a href="http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3" target="_blank">http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3</a></div>
<div><a href="http://www.erlang.org/doc/man/global.html#re_register_name-2" target="_blank">http://www.erlang.org/doc/man/global.html#re_register_name-2</a></div><div><br></div><div>BR,</div><div>Ulf W</div><div><div></div>
<div class="h5"><br><div><div>On 31 Mar 2011, at 14:36, Jeroen Koops wrote:</div><br><blockquote type="cite">Hi Ulf,<div><br></div><div>Thanks for the explanation, the message-flow is now pretty clear. </div><div><br></div>
<div>Actually, the reason I was looking into it was to see if I could come up with a patch that makes it possible to configure an alternative takeover behaviour, since I find it inconvenient that during a takeover the application is running on two nodes at the same time for a while, which leads to clashes with globally registered names, and problems with processes that assume they have exclusive access to data. It would be nice to be able to configure takeover behaviour in such a way that the application is only started on the node that is taking over <i>after </i>it has been stopped on the original node.</div>
<div><br></div><div> However, I've come to the conclusion that the whole application distribution is way too complex to start poking around, so instead I now use a mechanism where, whenever an application is started with { takeover, OtherNode } as StartType, it simply terminates all the children of the application's top-level supervisor on OtherNode before proceeding with the startup. This is a bit clunky, and a feel a bit sorry for the application on OtherNode being left behind as an empty shell, with all its children killed, waiting for the application controller to put it out of its misery, but it gets the job done...</div>
<div><br><br><div class="gmail_quote">On Wed, Mar 30, 2011 at 4:00 PM, Ulf Wiger <span dir="ltr"><<a href="mailto:ulf.wiger@erlang-solutions.com" target="_blank">ulf.wiger@erlang-solutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><br></div><div>Hi Jeroen,</div><div><br></div><div>This is a bit tricky, and can be very confusing for someone trying to build their own cluster controller. :)</div><div><br></div><div>
The stopping of the application is done by the local application_controller when it is told that the application is running somewhere else. In other words, dist_ac kindly forwards the information to the local AC, and it is the local AC that in this particular case takes responsibility for 'knowing' that the local instance should be stopped.</div>
<div><br></div><div>See application_controller:handle_application_started/3</div><div><br></div><div><a href="https://github.com/erlang/otp/blob/OTP_R14B02/lib/kernel/src/application_controller.erl#L913" target="_blank">https://github.com/erlang/otp/blob/OTP_R14B02/lib/kernel/src/application_controller.erl#L913</a></div>
<div><br></div><div>(The dirty deed actually happens on line 946).</div><div><br></div><div>The reason I know this, is that in the AXD 301, we ran multiple instances of some applications, distributed across several mated pairs - each instance having its own standby node. This can be done by writing an own distributed AC, but it has to be smart enough to know _when_ to forward the {started, Node} status to the local AC; if an instance was running locally - and _was supposed to_ do so (i.e. not involved in a takeover), the distributed AC had to suppress this message.</div>
<div><br></div><div>BR,</div><div>Ulf W</div><br><div><div><div></div><div><div>On 30 Mar 2011, at 14:11, Jeroen Koops wrote:</div><br></div></div><blockquote type="cite"><div><div></div><div>Hi,<br>
<br>I'm trying to find out how the distributed application controller works internally. I'm especially interested in the implementation of an application takeover.<br><br>In case an application runs on node A, and is taken over by node B, what should happen is that it is first started on node B, so that there are two instances of the application running simultaneously for a brief period of time, and then stopped on node A.<br>
<br>However, I cannot figure out where this stopping happens in <font face="'courier new', monospace">dist_ac.erl</font>. If I understand correctly, this should happen in response to a <font face="'courier new', monospace">ac_application_run</font> message from the application_controller. This message is received by the <font face="'courier new', monospace">dist_ac</font> on node B, and a <font face="'courier new', monospace">dist_ac_app_started</font> message is then broadcast to the <font face="'courier new', monospace">dist_ac</font>s on all connected nodes. The <font face="'courier new', monospace">dist_ac</font> of node A receives this message, notices that the application is still running locally, and decides to shut down the application on its own node -- at least that is what the comments say (<font face="'courier new', monospace">dist_ac.erl</font>, line 529):<br>
<br><font face="'courier new', monospace">%% Another node tookover from me; stop my application<br>%% and update the running list.</font><br><br>But all I can see is that the <font face="'courier new', monospace">dist_ac</font>'s list of applications is updated to indicate that the application is no longer running locally -- I cannot find where the application_controller is instructed to actually shutdown the application.<br>
<br>Can anyone point me in the right direction?<br><br>Thanks,<br><br></div></div>Jeroen
_______________________________________________<br>erlang-questions mailing list<br><a href="mailto:erlang-questions@erlang.org" target="_blank">erlang-questions@erlang.org</a><br><a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</blockquote></div><font color="#888888"><br><div>
<div>Ulf Wiger, CTO, Erlang Solutions, Ltd.</div><div><a href="http://erlang-solutions.com/" target="_blank">http://erlang-solutions.com</a></div><div><br></div><br>
</div>
<br></font></div></blockquote></div><br></div>
</blockquote></div><br><div>
<div>Ulf Wiger, CTO, Erlang Solutions, Ltd.</div><div><a href="http://erlang-solutions.com" target="_blank">http://erlang-solutions.com</a></div><div><br></div><br>
</div>
<br></div></div></div></blockquote></div><br></div>