[erlang-questions] non-trivial supervisor idioms? -- An open-source plea!

Ulf Wiger ulf.wiger@REDACTED
Wed Oct 27 10:17:43 CEST 2010 

Edmond,

One big problem for a company releasing large and complex, battle-tested software
as Open Source is that Ericsson - with a ~30% share of the global mobile infrastructure
market - is a juicy IPR target. Since it's entirely possible - nay, almost a certainty - that 
code of that size will infringe on some existing patents*, and given that it is a practical
impossibility to figure out which (unless you happen to be the owner of the patent, 
and know what to look for), it is unlikely to happen.

* most of which would probably not hold up in court, but challenging them in court
  is still a risky and expensive process - not least with huge opportunity cost.

There are a few feeble protections against being sued for patent infringement:

- Use a fringe technology. This invalidates many claims, as they are often fairly 
  language-specific.
- Do not search the patent database. This may seem absurd, but you probably 
  won't find anything anyway, but if you did search, you may eventually get sued
  for wilful infringement, which will increase the penalty significantly.
- Do not release your source code, so that it can be indexed and subjected to 
  automated search programs by patent raiders.
- Stay small and uninteresting, so no one will bother suing you.
- Keep a really big patent portfolio, so you can sue them right back.

Ericsson is not small and uninteresting, but they _can_ exercise caution in releasing
source code. Given this, it is actually no small matter that they maintain Erlang/OTP
in the open. :)

But hey, it's their decision. I'm not saying it's necessarily a bad idea - just guessing 
at some of the possible impediments.

BR,
Ulf W

On 27 Oct 2010, at 09:49, Edmond Begumisa wrote:

> That puts a nice perspective on things... obvious now I was trying to use included apps for something they were not designed for. But your story really got me thinking...
> 
> Sometimes I wish our friends at Ericsson would open-source one of their larger Erlang projects. Those of us who have NOT picked up Erlang from industrial environments have a problem of finding code we can reliably read for inspiration.
> 
> I (and I think many), heard about Erlang from somewhere, went and read Joe's book (a superb introduction which covers the 'spirit' of Erlang), identified with the problems Erlang tries to solve, maybe read Cesarini and Thompson as a follow up (an excellent reference.) The problem is no matter how good these books are, there's only so much a book can teach you. A book certainly cannot advise you on how to write a complete application or how to avoid writing programs that become unwieldy as they grow. The question is what next?
> 
> The obvious answer is to look at some open-source code. But when most open-source projects have learned Erlang the same way it becomes a case of the blind leading the blind. Problems that must have already been experienced and dealt with at Ericsson are repeated. A large production-ready open-source project written by battle-hardened experienced Erlang programmers would really fill in that void. Something big and sufficiently complex like a soft-switch or something.
> 
> - Edmond -
> 
> 
> On Tue, 28 Sep 2010 17:07:16 +1000, Ulf Wiger <ulf.wiger@REDACTED> wrote:
> 
>> 
>> On 27 Sep 2010, at 20:05, Edmond Begumisa wrote:
>> 
>>> Hi again Ulf,
>>> 
>>> It's great to get 'the guy' on this subject online so I'm going to take full advantage and ask two more questions that have been dogging me...
>>> 
>>> Firstly, is there an open-source project you know of that uses included-applications and/or start phases properly that I could take a peek at? Maybe in OTP source itself?
>> 
>> Off the top of my head, I really can't think of any. :)
>> 
>> The area where start phases really come in handy is when your application needs
>> to support failover/takeover behaviour. This is also when the StartType argument
>> is needed. One can implement takeover by writing a special start phase that instructs
>> the processes to take over processing from the other side. In general, it is best to do
>> this at a point where all processes have been started and are ready to process
>> incoming requests.
>> 
>> The initial reason for start phases was that the complex call-handling applications
>> at Ericsson had some pretty horrendous dependencies to sort out before they
>> could start accepting calls, and doing this work in the init function of the processes
>> simply wasn't feasible. Also, when a process dies in the init function, this is
>> interpreted as a start error, and the application start will fail, whereas individual
>> processes have proper supervision while they are responding to requests from
>> the start phase code (which runs in the application_starter process).
>> 
>> Included applications were mainly introduced since the same call-handling
>> applications needed to move as one during failover and takeover, and starting
>> a dozen or so top applications made that much more difficult. It was just too much
>> code and too many modules to integrate into one single application without one
>> more structuring layer.
>> 
>> Initially, I wrote some code that read .appSrc files in each sub-application and
>> integrated them into one larger application, using a top-level resource file - I
>> think it had the extension .appLm (as in load module - never mind; it made sense
>> at Ericsson, and it was so long ago that I may be remembering wrong). This was
>> later generalised by OTP into included_applications.
>> 
>> The O&M applications also had a problem during takeover: The snmp code
>> assumed that the snmp agent was locally registered on the same node, which
>> wasn't necessarily the case during the transition - either on the node taking over
>> or on the node where it ran before. We then created a wrapper application that
>> included all the O&M applications, and called the individual start functions for
>> each included app.
>> 
>> Later, we moved away from that, as we had to also support applications that
>> were written according to a different timeline, and therefore couldn't be integrated
>> the same way as our other apps. I came up with a solution for starting and stopping
>> included apps and plugging in their start phase hooks in the right places in the
>> startup flow, but for some reason people found it complicated... :)
>> 
>> The better solution was to make use of the fact that the application controller now
>> had a message passing interface for controlling the starting and stopping of apps.
>> We were already using this in our cluster controller, so we could extend it by
>> specifying distributed start dependencies and which applications needed to do
>> takeover in parallel. This way, the cluster controller knew in which order to move
>> applications during takeover, and in which order to terminate them, once migrated.
>> Unfortunately, all this code is proprietary. It's on my long list of things I'd like to do,
>> but that list just keeps growing, without much ever being removed from it...
>> 
>> A long time ago, I made a prototype (and sent to OTP) that introduced start phase
>> dependencies. This would IMHO make it much easier to specify dependencies
>> between applications. As an example, mnesia loads tables in the background, so
>> when the application:start() function returns, one cannot assume that tables are
>> loaded, and has to call mnesia:wait_for_tables() (which can time out, and has some
>> corner cases where tables will never be loaded without intervention - not that the
>> function itself will tell you when they occur). It might be better if mnesia had a
>> load_tables start phase, which other applications could depend on.
>> 
>> BR,
>> Ulf W
>> 
>>> 
>>> Secondly, I've always liked the idea of using included applications not necessarily for start phases but as a delayed/start-on-demand mechanism (taking advantage of the fact that included apps are automatically loaded but not started.) That is, manually calling application:start(foo) only if a particular feature of my app is used. But I have one query that made attempts for such use short-lived... the fact that an application can only be included by one other application. I think this limitation makes it harder to use included apps and start phases especially if you're using apps that are not in-house.
>>> 
>>> For example, lets say CouchDB starts using mnesia (ok that's dumb but...) and decide to start it up using start phases (and therefore add it as an included application in couch.app) Then I have my FunkyApp that's been using mnesia too as included application. I then decide to use CouchDB for a new funky feature of FunkyApp. Now things break because mnesia is being used by both FunkyApp and CouchDB. To fix this, I not only have to modify my in-house app I have to modify the out-house CouchDB too.
>>> 
>>> Is there an obvious fix to this I've been missing?
>>> 
>>> - Edmond -
>>> 
>>> On Tue, 28 Sep 2010 03:19:29 +1000, Ulf Wiger <ulf.wiger@REDACTED> wrote:
>>> 
>>> 
>>> On 27 Sep 2010, at 18:14, Edmond Begumisa wrote:
>>> 
>>> Ulf,
>>> 
>>> I've been doing such initialisation in the init function of a worker manager process. Using Daniel's example, I might have a gen_server child of the main supervisor called db_mgr and set up the mnesia schema in db_mgr:init
>>> 
>>> Have I been doing the 'wrong' thing OTP-wise?
>>> 
>>> Not necessarily, but my personal preference is to cleanly separate setup code
>>> from application startup. This is in part because I used to work on a very complex
>>> product, where the setup was decidedly non-trivial, and the startup process had
>>> to be optimised in several steps.
>>> 
>>> Still, even there, I believe that the setup logic was bootstrapped into the startup
>>> phase, but the code was still kept cleanly separated. The only thing that was
>>> part of the startup was a simple check to see if the setup code had been run.
>>> 
>>> BR,
>>> Ulf W
>>> 
>>> 
>>> 
>>> - Edmond  -
>>> 
>>> On Tue, 28 Sep 2010 00:31:47 +1000, Ulf Wiger <ulf.wiger@REDACTED> wrote:
>>> 
>>> On 27/09/2010 16:15, Daniel Goertzen wrote:
>>> I've read the documentation on supervision and have seen a few tutorials,
>>> but they don't seem to move beyond the core concepts.  For example, what
>>> happens if you want to check and optionally setup an mnesia schema during
>>> startup...where should this code go?  In the supervisor init() or
>>> start_link() function?  Should I have my supervisor create a worker process
>>> whole sole job is to do this kind of setup and then dynamically add other
>>> workers (or supervisors) to the supervisor with start_child()?
>>> 
>>> I strongly recommend doing that sort of thing in a separate procedure,
>>> rather than in the startup phase.
>>> 
>>> If you want your application to be able to bootstrap itself, I would
>>> suggest that you either:
>>> 
>>> - create a special application that runs before your other apps,
>>> and verifies that the installation is ok. To this end, it might be
>>> useful to know that you can pre-sort the .rel file. The systools lib
>>> will only change the sort order if needed to respect start
>>> dependencies.
>>> - Introduce start_phases, then do minimal work in the init function,
>>> and push the rest to functions that are called from start phase
>>> hooks. This also has the advantage that you know that your processes
>>> are all started and ready to respond during the init phase.
>>> 
>>> Start phases are documented in
>>> http://www.erlang.org/doc/apps/kernel/application.html#Module:start_phase-3
>>> 
>>> BR,
>>> Ulf W
>>> 
>>> 
>>> 
>>> --
>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>> 
>>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>>> http://erlang-solutions.com
>>> 
>>> 
>>> 
>>> 
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>> 
>>> 
>>> 
>>> --
>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>> 
>> Ulf Wiger, CTO, Erlang Solutions, Ltd.
>> http://erlang-solutions.com
>> 
>> 
>> 
>> 
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>> 
> 
> 
> -- 
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com

More information about the erlang-questions mailing list