[erlang-questions] Embedded vs Interactive - Why embedded?

Michael Truog <>
Thu Mar 10 21:43:05 CET 2016

On 03/10/2016 12:09 PM, Ryan wrote:
> On 03/09/2016 09:11 PM, Michael Truog wrote:
>> My understanding is that embedded is preferred for any production use, to make sure the system will fail-fast upon startup if there are any problems loading dependencies.  The best time to have something fail is when it first starts, due to its lifetime being undefined and possibly infinite (ignoring heat death in the universe, and other natural disasters that should be written out of legal agreements :-). Ideally this concept is extended into the runtime of the server, into initialization and configuration source code, to make sure the server can fail-fast upon startup when the server is misconfigured, rather than waiting an arbitrary number of hours or days to find out that a problem exists. This approach helps to avoid a reliance on a fire-fighting mentality that becomes dependant on monitoring for feedback on a system's health due to the source code being a potentially unknowable black-box for some organizational reason.
> This is a really good point, so let's clarify this. As I understand it, the difference between the two modes is that in embedded mode, all of the modules declared in your boot script are loaded at startup. To your point, if any modules happened to be missing, then startup would fail at this point. That's good. We like early failure. As far as I can tell, though, that's the only difference between embedded and interactive. After that initial "load all the modules", the next step would be to start all of your required applications. Whether by command line flag or boot script, that's going to progress the same way in either mode, assuming all the same modules are present. You talk about a initialization and configuration failing. Won't that cause identical issues in either mode, regardless of when the modules are loaded? I think all we're talking about here is whether code is loaded eagerly or lazily, and starting applications or processes works the same in either case.

The embedded/interactive functionality really is focused on module loading either at startup or lazily, as you have described.  I only mentioned initialization and configuration source code, due to how this fail-fast concept can be applied to source code.  While it may seem that the embedded/interactive choice is not an important one, with execution generally happening in the same way, it can be important due to some code paths being infrequent and problems with the dependencies like modules with the same name (and unfortunately sometimes it then depends on the search directory order, which can lead to problems during execution that are counter-intuitive).

>> The interactive mode helps when testing, since things are loaded automatically and you can have less concern about the dependencies, since you are actively creating or modifying the dependencies.  In production, you want an iron fist's control on all the dependencies, to make sure everything is repeatable and the service is stable, otherwise the development is pursuing stability in a serious way.  So, that means that production usage should demand that all the dependencies come from a specific place at a specific version that is known and tracked, to be easily replicated, without any ambiguity.
> Again, I completely agree that you want everything set in stone for production. I have a two-fold reply to your points.
> First, IMHO running two different ways in dev/test vs production only *increases* the chance that errors will slip in. Part of my quest here is to make it so that dev/test environments behave as similarly to production as possible so as to eliminate issues before they make it that far.

I meant using interactive mode for manual usage of the Erlang shell, not real testing of a release.  Only using interactive mode for development testing of random segments of Erlang source code.  Even that usage of interactive mode can be problematic due to the undocumented differences between the Erlang shell execution and normal Erlang module execution.  So, all releases for testing and production should be real releases running in embedded mode.  The interactive mode just helps you quickly use the Erlang shell to check stuff.

> Second, as to having complete control over dependencies in production, I don't think I'm talking about dependency management, just about code loading. What dependencies get deployed where is part of the release process. That's different from the embedded/interactive discussion, isn't it? In fact, I know that running embedded doesn't protect you from dependency problems because just a few months ago, we had a production release go bad because of a dependency issue. Again, we currently run *embedded mode* in production. The problem was that some application wasn't explicitly declared as a dependency in the right .app file, and it wasn't included in the .rel file, so it didn't get bundled into the release. The odd thing was that a few modules from that application *did* make it into the release. I'm not sure whether it was rebar or OTP that was responsible for that, but it made for a very confusing situation, even for a couple of fairly experienced devs who have been on this 
> project for 2 years now. This same version ran *perfectly* in dev and test. It was only in production that the bug manifested.

That can be weird.  I know there can be problems with reltool including dependencies that are not dependencies of the main application, due to xref being used internally by reltool instead of just looking at the .app dependencies.  That only affects using applications dynamically though, and doing that is uncommon. Normally all the Erlang applications are part of a static hierarchy and you only use a single boot file during the lifetime of the Erlang VM.

> Now, let me give a strong caveat that it's possible that we're misunderstanding something about how our release gets built, since it was someone else who wrote that part of the code, and he's no longer around. My point, though, is that dependency management is a build-time problem, isn't it? When you build a project, whether for testing or for production, that's when the right dependencies should be put in place. If you wait until runtime, you're too late. Stuff will crash.

Yes, release building is a build-time concern, but making sure the release is ran in a dependable way is what relates to the embedded/interactive mode decision.  Always using the embedded mode when a release is ran will help make sure the release is executed dependably.  You may have the initial startup cost of loading all your modules due to embedded mode but that delay is very small with the Erlang VM and I have never seen it as a problem (even with an ARM and slow SSD memory).

> Thanks very much for your reply. I really want to understand the basic issues here. Please let me know if I'm totally off base here. I have many years of dev experience, but not much with Erlang, so I'm still trying to find my way around.
> Ryan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20160310/d7d10535/attachment.html>

More information about the erlang-questions mailing list