[erlang-questions] sys.config and OTP-4867

[Felix Gallo]

I've had some success with the following automatable steps:

1.  Build a node in the usual way
2.  Replace its sys.config with a canary unbootable "you screwed up"
sys.config
3.  Commit that node as an artifact to porcelain artifact repository of
choice, i.e. git
4.  Commit per-environment sys.config setups separately
5.  As part of build process, pull the node artifact, marry it with
appropriately environmented and versioned sys.config

Beyond the raw confusion and complexity issues you note, there's also
pragmatically the question of live upgrades; if app x was booted with
sys.config a, and you upgrade app y in the same node that wants sys.config
b, then the unitary nature of sys.config starts to feel a little difficult.

Fundamentally the 12 factor app principles (http://12factor.net/) which I
believe you may be obliquely referencing are still themselves pretty new,
and erlang hasn't made the jump over there yet. Some of the ideas (e.g.
http://12factor.net/config, 'store everything in environment variables')
are super great for webapps, but webapps are normally written in
single-threaded memory-leaky buggy dynamic languages and so the expectation
that they'll be constantly murdered by their users or operators or
frameworks themselves and subsequently restarted is built into the
thinking.  Nodes, which stay up until armageddon happens, probably need a
more dynamic and mutable configuration system than that, because it may not
be seconds or minutes or indeed years between unix-process-restarts
refreshing the env vars.

Since misconfiguration is probably the #1 cause of outages at scale (
http://danluu.com/postmortem-lessons/, and extrapolating from my own
extensive experience killing clusters and data centers), this is a really
great area of exploration for some enterprising and thoughtful erlanger.

F.

On Thu, Aug 20, 2015 at 6:33 AM, Jesper Louis Andersen <
jesper.louis.andersen@REDACTED> wrote:

> In the config(5) man page we have a sys.config section:
>
> http://www.erlang.org/doc/man/config.html
>
> Which allows us to merge a file into the existing sys.config:
>
> [{myapp,[{par1,val1},{par2,val2}]},
>  "/home/user/myconfig"].
>
> This is a very neat trick, but if we read the code in
> `application_controller` there is a reference to OTP-4867 and this is only
> enabled for files named `sys.config`. This has bitten me several times,
> when people end up naming
> their configuration file differently. What is the reasoning behind
> limiting this to sys.config? And would it be possible to relax the naming
> constraint such that there is a set of files which gets file expansion?
>
> The reason I'm asking is because the whole game of configuring an
> application is a complex afair. Some environments are able to write files,
> whereas others are relying on the OS environment variables for certain
> settings. And yet others would like to be able to point to a file-chain
> which gets progressively merged into the configuration, like in sys.config.
>
> In some environments, sys.config is run through a preprocessor and its
> contents are replaced by writing a temporary file next to sys.config. But
> once loaded, this newly temp file is not amenable to OTP-4867 substitution
> :/ And writing in the other direction, from a stanza file into sys.config
> also seems somewhat futile.
>
> Clearly there were a lot of thought put into this configuration inside
> Ericsson, so in particular, how is a large Erlang system configured in a
> painless way? Specifically, how does one go about procuring a release of
> the software which doesn't change its checksum but can run in several
> environments: staging, user-acceptance-testing, production and contains
> *no* secret keys as part of its build but reads these from outside (e.g.,
> the environmental "context" in which it has been deployed?)
>
> --
> J.
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150820/b4a53ed5/attachment.htm>