[erlang-questions] Re: how to scale into the cloud using process? example computing simple average

[Ulf Wiger]

----- "Kid Erlang" <kiderlang@REDACTED> wrote:

> hi everyone.  i am still confused?  is there a good book on
> ERLANG to read on how to do multi process cloud scaling?
> 
> i do not understand your answers

Apologies for the somewhat existential rants.

You should read Joe Armstrong's book, which explains a lot 
of things, including how to scale with Distributed Erlang 
and multicore.

If you use Distributed Erlang, that is, many Erlang nodes 
connected in the most natural Erlang way, you will simply 
use processes and message passing. When using sockets, you 
can write a module similar to the rpc module, but which 
calls term_to_binary(Data) before sending the request to
the socket, and binary_to_term(Bin) on the response coming
back (and vice versa on the other end). It is reasonably 
straightforward. Joe's book has a chapter (ch 14) on 
socket programming, which pretty much spells out how to 
do this.

One of the new problems with clouds is how host names can
come and go, making it difficult to have the kind of static
configuration of the members as one usually has in a cluster
(where one usually has full control over the IP addresses,
host names, etc.)

Joel Reymont used to have a blog article about setting up
mnesia for EC2, which AFAIR addressed these issues, but 
the article seems to have rotted away.

I found this:
http://blog.onclearlake.ca/2008/03/setup-ec2-instance-with-erlang-this.html


> i want to scale into the cloud to generate bigger sets of
> numbers and be able to run multiple copies of my scripts which
> already run in a collection.

In your example, you are basically doing addition. You should
be aware that this form of work cannot be sped up even by 
parallelizing on multicore. Just running through the list and
spawning a process (or even sending a message) is more work than
just performing the addition.

I showed an example of that in my Erlang Factory presentation:
http://www.erlang-factory.com/upload/presentations/56/UlfWiger_ErlangMulticoreEF.pdf
(slide 9 shows some simple benchmark figures on two different 
types of jobs being parallelized.)

BR,
Ulf W
-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd.
http://www.erlang-consulting.com