[erlang-questions] Why have a supervisor behaviour?

[Christopher Phillips]

>
>
> Message: 17
> Date: Thu, 21 May 2015 16:32:40 -0400
> From: Fred Hebert <mononcqc@REDACTED>
> To: Roger Lipscombe <roger@REDACTED>
> Cc: "erlang-questions@REDACTED" <erlang-questions@REDACTED>
> Subject: Re: [erlang-questions] Why have a supervisor behaviour?
> Message-ID: <20150521203238.GA52846@REDACTED>
> Content-Type: text/plain; charset=us-ascii; format=flowed
>
> On 05/21, Roger Lipscombe wrote:
> >I need delayed _restart_. Is this what Jesper refers to when he talks
> >about "a delay_manager"? Such that init queries that and then
> >might/might not delay?
>
> That's a classic question, and one I started answering differently.
> Requiring a timeout in your supervisor rebooting function means that you
> are letting things crash or restart for the wrong reason.
>
> The thing is, it's all about the guarantees[1]. In a nutshell, a
> supervisor should exit on any error, and ideally bring you back to a
> known, stable state.
>
> So of course all expected or unexpected errors should be able to bring
> you back to that state properly, specifically transient errors.
>
> But the distinction is that because supervisors boot synchronously for
> all apps, they also represent a chain of dependencies of what should be
> available to all processes started *after* them.
>
> That's why failure modes such as 'one for all' or 'rest for one' exist.
> They allow you to specify that the processes there are related to each
> other in ways that their death violates some guarantee of invariant in
> the system and that the only good way to restart is by refreshing all of
> them.
>
> In a nutshell, if you expect disconnections or event that require a
> backoff to happen frequently enough they are to be expected by the
> processes depending on yours, then that connection or that event is not
> a thing that should take place in your process' init function. Otherwise
> you're indirectly stating that without this thing working, the system
> should just not boot.
>
> See the example in [2] for an idea of how to respect this. This does not
> change the code in any major way, but moves function calls around to
> properly respect these semantics.
>
> My position is that this isn't a problem with supervisors' interface,
> but in how they are being use and what they mean for your system. I know
> this is not the most helpful response, but oh well.
>
>
> [1]: http://ferd.ca/it-s-about-the-guarantees.html
> [2]: http://www.erlang-in-anger.com, section 2.2.3
>
>
> I wanted to add, every time I've seen this pattern brought up (a
supervisor with a delayed restart), it's been due to something on the
network has become unavailable, or overloaded to where it can't respond in
a reasonable amount of time, that sort of thing, and the developers
involved were seeing restart limits being hit and the system coming down,
even though there was every reason to expect the resource to become
available again, in time.

If that's the case for you, those types of issues would likely be better
addressed using a circuit breaker or similar[1] than a custom supervisor,
for the reasons Fred mentions.

In general, having this thing be unavailable is either due to bad internal
state (in which case a supervisor can help you), or something external to
your program (in which case the supervisor can't), and in the latter case
you should be thinking about how the system should behave when it's
unavailable (since it would be unavailable even with a delayed supervisor
in any case). Making it a clean response that means "this resource is not
available" allows you to address it in a well defined way, rather than
having a child process that may or may not be there being called from
elsewhere in your system.

[1]: https://github.com/jlouis/fuse
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150522/2715f7bd/attachment.htm>