<div dir="ltr"><div>Hi Andreas,<br><br></div><div>This is due to the failover mechanism you are using, ie marking the app as 'distributed' in sys.config.<br></div><div>This makes the kernel start the dist_ac controller to manage your app.<br>
Unfortunately, the dist_ac controller is looking for nodedown, not app dieing.<br>Reading carefully on one of the pages your writeup links to (<a href="http://learnyousomeerlang.com/distributed-otp-applications">http://learnyousomeerlang.com/distributed-otp-applications</a>)<br>
<br>"The only thing they do is wait for the node of the running application
to die. This means that when the node that runs the app dies, another
node starts running it instead.</div><div><br>The dist_ac controller is also susceptible to net splits. If the
network connection between your 2 nodes goes down, the app will be
running on both nodes. Once the network heals again, dist_ac doesn't
know what to do about it. I see in your example both nodes are on the
same machine, so you aren't susceptible to this limitation.<br><br></div><div>I got bit by these problems a while back. I eventually had to roll my own distributed app controller based on gen_leader: <a href="https://github.com/garret-smith/gen_leader_revival">https://github.com/garret-smith/gen_leader_revival</a><br>
</div><div><br></div><div>-Garret Smith<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 11, 2013 at 4:31 AM, Andreas Pauley <span dir="ltr"><<a href="mailto:apauley@gmail.com" target="_blank">apauley@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi everyone,<br>
<br>
I've made a demo app of mine distributed to test failover and<br>
takeover, after reading the "Distributed OTP Applications" chapter in<br>
Learn you some Erlang.<br>
<br>
The failover and takeover works great if I kill the running beam (eg.<br>
with kill -9).<br>
<br>
However, I tried sending kill signals to the Pids of both my<br>
application behavior and the top supervisor that gets started by the<br>
application.<br>
This crashes the VM, but failover does not happen.<br>
<br>
Is this unsupported, or should I do something to enable failover in<br>
this scenario?<br>
<br>
I've done a more complete writeup with code and output here:<br>
<a href="https://github.com/apauley/dark-overlord#when-processes-die-a-guide-to-the-afterlife" target="_blank">https://github.com/apauley/dark-overlord#when-processes-die-a-guide-to-the-afterlife</a><br>
<br>
But in a nutshell, I crash the VM with the commands below, and then<br>
automatic failover to my second node does not happen:<br>
<br>
$ ./rel/overlord/bin/overlord console<br>
Erlang R15B03 (erts-5.9.3.1) [source] [64-bit] [smp:8:8]<br>
[async-threads:0] [hipe] [kernel-poll:false] [dtrace]<br>
<br>
14:03:20.581 <a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a> [overlord_app] <0.56.0> || Starting app: normal<br>
14:03:20.582 <a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a> [hypnosponge_sup] <0.57.0> || Hello<br>
from the hypnosponge supervisor<br>
(<a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a>)1> Sup = pid(0, 57, 0).<br>
<0.57.0><br>
(<a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a>)2> exit(Sup, kill).<br>
<br>
=ERROR REPORT==== 11-Feb-2013::14:04:46 ===<br>
** Generic server minion_supersup terminating<br>
** Last message in was {'EXIT',<0.57.0>,killed}<br>
** When Server state == {state,<br>
{local,minion_supersup},<br>
simple_one_for_one,<br>
[{child,undefined,minion_makeshift_sup,<br>
{minion_makeshift_sup,start_link,[]},<br>
temporary,5000,worker,<br>
[minion_makeshift_sup]}],<br>
undefined,1,3,[],minion_supersup,[]}<br>
** Reason for termination ==<br>
** killed<br>
true<br>
(<a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a>)3><br>
=INFO REPORT==== 11-Feb-2013::14:04:46 ===<br>
application: overlord<br>
exited: killed<br>
type: permanent<br>
<br>
(<a href="mailto:overlord@127.0.0.1">overlord@127.0.0.1</a>)3> {"Kernel pid<br>
terminated",application_controller,"{application_terminated,overlord,killed}"}<br>
<br>
Crash dump was written to: erl_crash.dump<br>
Kernel pid terminated (application_controller)<br>
({application_terminated,overlord,killed})<br>
<span class=""><font color="#888888"><br>
--<br>
<a href="http://pauley.org.za/" target="_blank">http://pauley.org.za/</a><br>
<a href="http://twitter.com/apauley" target="_blank">http://twitter.com/apauley</a><br>
<a href="http://www.meetup.com/lambda-luminaries/" target="_blank">http://www.meetup.com/lambda-luminaries/</a><br>
_______________________________________________<br>
erlang-questions mailing list<br>
<a href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-questions" target="_blank">http://erlang.org/mailman/listinfo/erlang-questions</a><br>
</font></span></blockquote></div><br></div></div>