[erlang-questions] Dependencies, included applications and supervision

[Michael Truog]

Comments below:

On 06/25/2014 01:32 PM, Jay Nelson wrote:
> I am a strong advocate of using included_applications, but even a stronger advocate
> of using “least constraint” architectural choices.
>
> If you choose to implement your application in a manner compatible with the included
> application rules (no initialization before root supervisor starts, only one root supervisor
> called directly from application:start function), then it is possible for someone else to
> include your application under their own supervisor hierarchy.
>
> If you choose not to follow these rules, then you have chosen to not allow anyone else
> to supervise your application from within theirs without access to your source and copy
> and paste maintenance whenever you release a new version.
>
> The latter is restricting the choice of future integration.
Normally, included_applications is used for supervisors that were not started as part of their application, or perhaps with an application that didn't provide a root supervisor.  When you are starting processes for a separate application, you are breaking encapsulation that Erlang/OTP applications can provide (problem 1). The other concern with using included_applications is that it can make supervision hierarchies larger (more levels in the tree) which can make it more difficult to have MaxR and MaxT settings that allow the whole application to fail when errors occur (problem 2).  So, using included_applications normally would be contrary to allowing applications to fail-fast.  included_applications is only a way to integrate with Erlang applications that are not using OTP (i.e., not using an application behaviour as it was intended), and it shows a design problem in the application that must be used in this way.

>
> When you run “sibling applications” three things happen:
>
> 1) You must choose whether application death is ignored, or whether it will crash
> the entire node — there is no in between.
>
> 2) You have to sprinkle ‘ensure_started’ calls throughout your code if there is a
> dependency. And trust me, you can never be completely assured that it is started
> at any given moment, ensure_started is yet another race condition.
I have some code at https://github.com/okeuday/reltool_util that does more than 'ensure_started' is able to do, since 'ensure_started' doesn't choose to handle some of the loading issues when trying to start or stop applications dynamically.  This code also is able to start a release dynamically based on the release's script file.  I think more dynamic usage of Erlang is important and that it is good to avoid having a single release that can only be provided on the command line.  So, making stuff more dynamic is advantageous, not making it less dynamic by having a single static hierarchy of processes.  I understand you are concerned about race conditions, which is a valid concern, but with proper timeout usage you can anticipate startup delays due to initialization.  If race conditions are still created, then Erlang processes from separate applications should be merged to make the startup deterministic while encapsulating hard dependencies that might otherwise cause failures 
due to race conditions.  I don't think included_applications will be required to merge the applications because the race condition problems would be in code you control.

>
> 3) You cannot guarantee start ordering and you may have unexpected race
> conditions if you try to roll your own restart mechanism (every combination of
> single or multiple application failure must be accounted for with every interleaving
> of application restart).
I agree that not having a dependable strict ordering of application startup can make some problems ambiguous, which is a concern, when startup is determined automatically based on dependencies and the result of release generation.  If you use a CloudI configuration file (http://cloudi.org), all services are listed in the order they are started, which is much easier to reason about when problems occur with CloudI services.  Internal CloudI services are services written in Erlang which can be (but are not required to be) Erlang applications, so they can start Erlang applications dynamically outside of a release, which provides more dynamic behavior than is normally available within Erlang.

>
> As Max said, you can take a hammer to it and force all application deaths to crash
> your node, and just ensure you always have multiple nodes, and a healthy load
> balancer and guarantee that there are no hiccups with long restart times.
>
> An application is really just a registry in an ets table with environment associated,
> and some thin semantics about starting. It is a semantic container for a supervisor
> hierarchy that provides a set of related functionality delivering a single service.
> I like to integrate single services into a larger hierarchy with ordering and restart
> semantics that I control. It is a more structured way to integrate erlang applications,
> by removing one container and placing more than one root supervisor into a new
> container.
>
> I think a better future is to have sibling services and the ability to have services
> degrade or go down, rather than a strict hierarchical supervision tree. But given
> only a supervision hierarchy, there is no reason to write a library or an application
> that precludes using the included_applications option. It should be the choice of
> the integrator how to organize (and deal with the consequences) of their own
> application server.
CloudI provides a better way of doing this.  Examples if you are interested for Erlang development and enforcing ordering are https://github.com/CloudI/CloudI/tree/master/examples/hello_world1#purpose and https://github.com/CloudI/CloudI/tree/master/examples/hello_world5#purpose .

>
> jay
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>