[erlang-questions] Erlang on supercomputers

[Ramon Diaz-Uriarte]

On 4/4/07, Thomas Lindgren <thomasl_erlang@REDACTED> wrote:
>
> --- Olivier BOUDEVILLE <olivier.boudeville@REDACTED>
> wrote:
>
> > Hi,
> >
> > first, sorry if the question is irrelevant, I am by
> > no means an HPC or
> > Erlang expert.
> >
> > I was wondering whether, for the simulation of a
> > large-scale distributed
> > system, one could use Erlang on a supercomputer such
> > as an IBM's
> > BlueGene/L (see
> > http://en.wikipedia.org/wiki/Blue_Gene for a
> > description).
> ...
> > Knowing that each processor is a PowerPC (and that
> > Erlang works on Mac)
> > running a very minimal (very light-weight,
> > mono-process) GNU/Linux OS with
> > limited POSIX support,  what I imagine is that it
> > would be feasible to run
> > on each of the, say, 4000 nodes, a stripped-down
> > Erlang interpreter that
> > would schedule itself (without much support of the
> > OS, no light-weight
> > threads) numerous interacting Erlang processes that
> > could communicate
> > preferably with other processes on the same node or,
> > otherwise, with
> > processes running on close nodes.
> >
> > Is having Erlang running on these architectures
> > technically possible with
> > a limited porting effort ? If yes, am I correct to
> > think that, provided we
> > manage to place cleverly the concurrent processes
> > (the more two of them
> > are interacting, the "closest" they are in the
> > communication topology, a
> > three-dimensional torus), we could in some way take
> > advantage of the
> > message-passing nature of Erlang to use (even
> > less-than-optimally) the
> > processing power of these machines ?
>
> As long as you're running some Unix-flavour, porting
> should be fairly straightforward. The regular Erlang
> VM is not excessively large either, so you could start
> out by just getting Erlang/OTP running and then pare
> things down.
>
> The snags I can see are:
>
> 1. Distributed Erlang could have scalability problems,
> since it by default uses a mesh of nodes the last time
> I looked. You may have to read up on "hidden nodes" to
> get this right. (I haven't studied this issue
> recently, however.)
>
> 2. BlueGene is basically a number cruncher, isn't it?
> While it's not hopeless to do that in Erlang, you
> probably want to implement the serious numerics in
> another language and link them up. Several methods are
> available.
>


I actually have a not unrelated problem (distributing number crunching
computations on a cluster). I was under the impression that this might
be a very reasonable way to go: have Erlang take care of distribution,
etc, and do the number crunching in another language (e.g., C). Any
specific catch?


> MPI: I'm not aware of any existing implementations.
> You can probably implement the MPI protocol in Erlang

I must be missing something: having Erlang, why would one want to
implement the MPI protocol? In my case, I am actually trying to
substitute code I have that uses MPI by the scheme above; for
instance, this ought to give me fault tolerance with a lot less cost
than what I am currently doing (checking MPI did not crash, restarting
MPI, etc, etc, plus lots of very cluster-dependent hacks).

Best,

R.




> though. The bit syntax is handy for that sort of
> thing. Or perhaps link with an existing
> implementation. I think there is a section on how to
> implement the Erlang distribution protocol in the
> docs.
>
> Best,
> Thomas
>
>
>
>
> ____________________________________________________________________________________
> The fish are biting.
> Get more visitors on your site using Yahoo! Search Marketing.
> http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>


-- 
Ramon Diaz-Uriarte
Statistical Computing Team
Structural Biology and Biocomputing Programme
Spanish National Cancer Centre (CNIO)
http://ligarto.org/rdiaz