[erlang-questions] Dynamic configuration of a running Erlang/OTP application

[Jachym Holecek]

Hi,

# Doug Fort, Consulting Programmer 2010-12-21:
> At SpiderOak, we have an application that involves clusters of 10
> nodes passing data files to each other.
> 
> Currently, if a node discovers that a peer is down (net_adm:ping
> returns pang), it stops sending to all peers. If one node is down, the
> cluster is down.
> 
> We propose to enable the sysadmins to notify nodes that one of their
> peers is down and to give an alternate destination for the missing
> peer.
> 
> Our current plan is to have a forwarding.conf file that the admins can
> scp to nodes in a cluster. The application will periodically poll this
> file for changes in the forwarding rules.
> 
> In the OTP world of applications that never shut down, others must
> have encountered this requirement. How can you change the
> configuration of a running process? (In an elegant way). We're looking
> for something that unix admins are comfortable with.

You could easily create an application, let's call it 'config', that
mirrors environment-related API of 'application' (get_env, set_env etc)
and is implemented as a Mnesia table using the underlying ETS table
directly (for read accesses) for speed.

Your admins can than easily change any config item at runtime from
shell (to_erl), something they'll undoubtedly love. To make it even
more admin friendly, you could come up with a web interface to the
config thing that would let them adjust some setting on given node
or all nodes in the cluster at once, using builtin RPC to distribute
the change (you could in theory make the config table replicated,
but that's not such a great idea on closer inspection since sometimes
settings are node-specific).

That said, requiring manual intervention on node failure sounds
weird, is there really no chance the system could cope by itself
(build failover behaviour into it and provide list of peers in
configuration, or somesuch)?

HTH,
	-- Jachym