[erlang-questions] how to scale into the cloud using process? example computing simple average

[Joe Armstrong]

So it sounds like a cloud is a dynamic pool to which you can independently add
workers and jobs. The jobs "just get done" and are allocated to the
best workers for
the job - there is a management interface so you can ask how the system is
doing, get statistics on the jobs etc.

What (Erlang) abstractions do we need?

Here's a suggestion for a cloud API

    cloud:add_job(Job) -> JobRef
          Job is a fun/0 that evaluates to result()
    cloud:status(JobRef) -> queued | {running, Where} | done | {suspended, Why}
          Query the status of the job
    cloud:result(JobRef) -> result().
           Can be called when status= done to get the result
    cloud:delete_job(JobRef) -> true
           remove all traces of the job

And the managment API

      cloud:add_node(Host, Port) -> NodeRef
      cloud:remove_node(NodeRef)
      cloud:nodes() -> [NodeRef]
      cloud:info() -> job_and_node_statistics()

I guess we might need a persistent storage abstraction to aid writing the
jobs. Lets assume the job refererences are used as keys into a persistent
fault-tolerent store

Then we need

     cloud:mutate_store(Fun/0) -> commit | fail

Fun0 is a function which can call

     cloud:store(JobRef, Key, Val)
     cloud:fetch(JobRef, key)

This is done with transaction semantics.

The API is intentionally minimal - How could this be implemented?
a layer on top of scalaris or Rabbit MQ?

Is this API complete?

Anybody want to build a prototype?

/Joe






On Wed, May 27, 2009 at 8:34 AM, Tony Arcieri <tony@REDACTED> wrote:
> On Tue, May 26, 2009 at 2:00 AM, Joe Armstrong <erlang@REDACTED> wrote:
>>
>> What is a cloud?
>
> It's basically the same as any other distributed environment, but there's a
> central authority you can go to and ask "I needs me more computars ktks" and
> it's like "ok thar you be bai" and you can get as many computars as you need
> to get job X done.
>
> That's really the only difference, except for the fact that when you're done
> the central authority is like "I need to eat the whole state of your
> instance in order to survive" and maybe you're like "no!" but it's like "I
> must I'm huuungary" and maybe with an environment like Amazon EC2 they have
> Elastic Block Store so you can rip out the guts of your state and stash them
> somewhere before the central authority beast eats up the state and resources
> of your instance and recycles them into new instances.
>
> --
> Tony Arcieri
> medioh.com
>