time syncronization in erlang cluster

[Per Hedeland]

Ulf Wiger <ulf.wiger@REDACTED> wrote:
>
>Of course there's a catch:
>
>If the computer running the Erlang node is set for
>any other timezone than Universal Time (UDT), the
>system clock will make jumps that erlang:now() will
>not be able to adjust to within reasonable time.

Hmm?? On a sensible OS (e.g. Unix or derivatives) there are no jumps,
since the system clock (and erlang:now()) always runs on UTC/GMT. The
time zone, including Daylight Savings Time changes (I assume those are
the jumps you're thinking of) is just a matter of how applications
display time to the user. Indeed, each user can have his own time zone
setting.

I don't know what erlang:now() does on Windows, but if it gives UTC
there too, it shouldn't be a problem that Windows stupidly runs the
system clock on local time and jumps it around on DST changes.

>On Unix systems, one creative way of getting around
>this (if, for any reason, you can't run the whole system
>in UDT) is to use os:setenv/2 and set the TZ environment
>variable to UDT inside the Erlang node.

And on Unix there is no need for it... However if you do run the system
on the local time zone, it's nice if e.g. time stamps in logs written by
Erlang change if the time zone setting is changed, and then you can use
the os:putenv trick to set TZ to the new value (assuming, of course,
that you have some way of knowing that it was changed and what the new
value is:-) instead of having to restart the Erlang node.

>Also, it's probably a good idea to have a background
>process sampling erlang:now() and time(), comparing the
>two. If the difference is too great (pick your definition
>of "too great"), it might be wise to restart the node.

Yes - what you potentially do need to be concerned with is a user making
a "big" change in the time "outside" the Erlang system. Whether this is
possible/"supported" depends on your application of course - if it's an
"appliance" or "embedded system" it's not unreasonable to require that
this only be done *through* Erlang (i.e. via the user interface of your
application), but if you have something that runs as an application
among others on a general-purpose OS, you may need to do some checking
like the above.

>On Fri, 9 Jan 2004 10:51:07 +0200, Primanathan Reddy [ MTN - Innovation 
>Centre ] <Reddy_P@REDACTED> wrote:
>
>> I'm going to try NTP.

And to get even more off-topic:-), I'll write a few words about how we
use NTP in Nortel's Erlang-based products. These run on Linux, but are
definitely in the "appliance" category - there is a custom CLI for
management (including all aspects of time), and if the user logs in as
root and messes about, the warranty is voided.:-)

These products can be clustered with up to 255 hosts in a cluster, and
have the concept of a "Management IP" (MIP) which migrates as needed
when hosts go down or are removed from the cluster, such that the user
can always reach the CLI by connecting to the MIP. This management is
all in Erlang of course.

For NTP, we "overload" the MIP concept insofar as the "MIP owner" is
always also the "NTP master". This means that all other hosts in the
cluster have a trivial NTP config, basically "server <MIP-address>".
The MIP owner is instead by default configured with its own local clock
(127.127.1.0 in NTP parlance) as a high-stratum (10 or more) "server".

This alone means that the hosts in the cluster are always synced to the
same time, and when the MIP migrates, only the new MIP-owner needs to
have its NTP config updated (the old MIP-owner is by definition not
there anymore). We brutally restart the NTP server (ntpd) on the new
MIP-owner at that point, but if you have high requirements on "smooth"
timekeeping, there are other possibilities with the tools included in
the reference NTP implementation.

In addition to this, the user can optionally configure one or more
"real" (external to the cluster) NTP servers. These are only added to
the MIP-owner config, keeping the config trivial on the other hosts and
reducing the traffic from the cluster to these servers. Since they have
a substantially lower stratum than that configured for the local clock,
ntpd on the MIP-owner will prefer them and instead correct its local
clock - but if the "real" servers become unavailable, it will fall back
to its (now time- and frequency-adjusted) local clock, and the cluster
will stay synced to a common time.

--Per Hedeland