[erlang-questions] best way to do live code update

[Joe Armstrong]

Live code upgrading is one of the things Erlang was designed for :-)

You have to distinguish between language primitives for live upgrade
and library support for live upgrade.

Erlang (the language) has very few mechanisms to support live code upgrade -
there is a minimal subset of mechanisms that you can use to build system that
can be upgraded on the fly.

The OTP libraries uses these mechanisms to make a particular form of live
upgrade - there are actually many different ways to do this the OTP
libraries represent one
such way.

Live upgrade can be programmed in "pure erlang" just as any other application.

If you code the inner loop of a server like this:

loop(Fun, State) ->
    receive
        {From, {rpc, Tag, Q}} ->
               {Reply, State1} = Fun(Q, State),
               From !  {Tag, Reply},
               loop(Fun, State1);
        {code_upgrade, Fun1} ->
               loop(Fun1, State)
     end.

Then you can get a simple form of code upgrade ... nothing fancy but it works

Then you have to co-ordinate things so that you know which processes to upgrade.

In a more realistic system you need notions of a "stable state" and of
state transformers

You should view versions of your systems as progressing from state1 to
state2 to state3
and so on.

Then you need transformation functions that take state N to state N+1 AND
(importantly) state N+1 to state N.

To make a state change in you system your put it into a stable state,
then roll the state forwards or backwards then resume operations.

Meta code for this is:

     suspend all relevant processes
     roll  to state I+1 or I-1
     resume all relevant processes

This is (basically) what the OTP release structure does

if you think about it upgrading/downgrading a system is pretty much like making
a fault-tolerant system (or a scable system)

Paradoxically I think code upgrade in a massively distributed system
is much easier than in a monolithic system.

In a large distributed system we stop node1 - change the code -
restart it, go to node
2 and so on. We also arrange that when node 1 is stopped some over
node can handle the traffic. Note that were node 1 to crash, some
other node should take over, so
we should have programmed this as part of the application.

In a single-node system things are much more difficult - no other nodes can take
over while we upgrade the code.

So as the number of nodes increases things get easier - other nodes
can take over
while we change the code.

In fact we can imagine that the system is always inconsistent - some
of the nodes
will never get the new upgrades.

What we've said about nodes is also true for processes.

Image a large system with thousands of nodes.

Such a system will probably always be in a mixture of versions.

When we do code upgrade some systems will be in the old state, others
in the new state.

When a crashed node is brought back on-line it might be several
versions out of date.

None of this matters if the protocols between the nodes can handle versions.

Let's suppose we have 100 nodes in state 34.

To upgrade to state 35 we suspend machine 1, upgrade it to state 35 then restart
it. Then we do machine 2 and so on.

While we are upgrading machine 1 we might move the traffic to machine 2.

How does code upgrade differ from a crash?

If machine 1 crashes what should happen? - another machine should take over.

So another way to do code upgrade might be

crash machine 1 (some other machine will take over)
upgrade the code in machine 1
bring machine 1 back on line

crash machine 2 ...
and so on

The point I want to make is that code-upgrade is a part of the application
which needs to be designed (like anything else) - it should to be
designed together with
the fail-over and scalability issues.

Personally I think the best way to do things is to view the system as being
permanently inconsistent and write code that always tries to make
things consistent.

We assume that we have N nodes and that they will all be running code
in different versions - if we assume that any node can crash and the
other nodes can take over
if we assume that we can add nodes to increase capacity.

Then we make an architecture for this - then to change code we just crash
nodes in a systematic manner (other nodes will take over) change the
code and restart them.

The key to this is using defined protocols between nodes which contain
version numbers.
Also the notion of stable state which can be replicated and survive a crash.

Basically the OTP behaviors hide and package mechanisms like these
into a convenient form.

The mechanisms themselves are pretty simple. You need to think about

   1) How to suspend the system
       and put it into a stable state
   2) How to replicate the stable state
   3) How to restart from a stable state
   4) How to detect failure
   5) How to upgrade and downgrade the stable state

Once you've done this you've done most of the work - then you can program this
in pure Erlang or use the OTP libraries - be warned there's a lot of
reading to do :-)

Cheers

/Joe


On 7/10/07, beepblip@REDACTED <beepblip@REDACTED> wrote:
> hi,
>
> what is the best way to do design a system for live code update?
> in theory, i would like to ship a tar/bundle/etc with all the necessary
> code. the end user should be able to load that while the system
> is running and then proceed as thogh nothing much has changed.
>
> this should be similar to a live code update that was done on the
> network gear that erlang ran on.
>
> how does one structure their project to accomplish this?
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>