The Workflow Grid (was RE: Brain Dump #1)

[Bob.Smart@REDACTED]

Joe Armstrong's call for unification of zope, 
jabber and blogger fits well with my personal 
project which I call the Workflow Grid. The 
followup pointers to other important work such 
as XUL have also been very useful to me.

The computing infrastructure that we use every  
day fails to deal well with:

    * Situations where the user has to deal 
      with computer and network activitiy that 
      takes a long time;
    * Activities involving multiple computers 
      and/or people that need to be synchronized, 
      or at least co-ordinated;
    * Cases where people want to deal with 
      various on-going activities at the same 
      time that have differing priorities and 
      time-delay characterisitics.

The Grid is the name for some new ideas to allow 
users, particularly scientists, to access a 
range of computing resources to get their work 
done. If you go looking at the Global Grid Forum 
site (www.ggf.org) and the globus site 
(www.globus.org) it isn't immediately obvious what 
those new ideas are. For me two stand out, and I am 
interested in other people's ideas on this. My two:

   1. Access to data, either simple files or 
      databases, is done via library calls rather 
      than the traditional access to files using 
      system calls to access mounted filesystems. 
      This makes it easier to use client credentials, 
      such as tickets or certificates, to provide 
      authorization.
   2. Limited proxy authentication. Authentication 
      is provided by certificates, but when you run 
      a remote job on a computer you don't entirely 
      trust you can give that job limited access for 
      a limited time to remote resources it might 
      need, such as access to remote files. This 
      fits well with the first point and makes it 
      practicable make serious use of a "grid" of 
      not fully trusted computers.

Naturally this could have been done better with 
Erlang, but its a bit late for that. The GGF has 
decided that grid communication should utilize Web 
Services technology. Version 3 of Globus (currently 
in alpha) partly complies with this.

Workflow systems handle the important situation 
where different people and software systems are 
involved over a period of time. The simplest case 
is a game such as chess: the players take turns to 
move. However both players have moves available that 
they can make at any time (resign, offer a draw). 
So one player is "expected to move", the other also 
"able to move". We use the term "activity" instead 
of "game" to indicate the breadth of possibilities.

We can see that users are likely to be interested in 
changes in state for activities thay are involved in, 
and particularly in changes that put them into 
"expected to move" mode. Traditionally e-mail has 
been the one system where one could interact 
asynchronously with other parties and be notified of 
incoming activity. This has led to the tendency to 
try to build workflow activities on top of e-mail, 
and more recently instant messaging like jabber. I 
would argue strongly that these messaging systems 
are not capable of doing this more general job. On
the other hand if you have a good workflow user 
interface then it is natural to add e-mail and 
instant messaging to that.

Workflow is particularly important for the Grid 
because it involves long running jobs involving 
multiple systems and it can involve multiple users 
working on the same problem. I believe that adding 
workflow as an outer layer is a mistake. Instead the 
correct answer is to have a strong but simple 
workflow service at the core of the Grid. Hence 
"The Workflow Grid". I have a demonstration workflow 
system in Erlang. It makes heavy use of mnesia, but 
hasn't been written as a general OTP application, 
though clearly it should be. It is quite a long way 
from what the Workflow Grid should be, bit it's a 
start.

This is the a rough document on the Workflow Grid
that was sent to Australian Grid Forum participants:
http://portal.hpc.csiro.au/~sma045/WG/workflowGrid.html

I've started a document describing how a Workflow
Grid environment would interact with existing important
services:
http://portal.hpc.csiro.au/~sma045/WG/WG-world.html

Those with a strong stomach could look at my old
workflow system (most of the code actually implements
a specific "game" which is a bridge bidding practice
system). It contains some relevant ideas that I 
would like to build on.
http://portal.hpc.csiro.au/~sma045/WG/asyncgame.tgz

To summarize: there is a whole area to be explored.
The needs of the server inner core are such that
Erlang/OTP is the ideal environment. The specific
areas mentioned by Joe Armstrong would be important
parts of any such system.

Bob Smart
CSIRO HPCCC