[erlang-questions] Keeping massive concurrency when interfacing with C

[CGS]

Sorry for interfering, but I would suggest CBLAS (BLAS rewritten in C)
instead of BLAS (which is written in FORTRAN). CBLAS is coming with GSL (GNU
Scientific Library) and it's thread safe (I tried it for a scientific Monte
Carlo simulation). GSL has the main advantage that it uses already tested
algorithms used by the scientific community (like CERN, FermiLab and so on)
where speed and precision is a must. But this is for really heavy linear
algebra.

Cheers,
CGS





On Wed, Oct 5, 2011 at 10:24 AM, Peer Stritzinger <peerst@REDACTED> wrote:

> On Wed, Oct 5, 2011 at 12:17 AM, Richard O'Keefe <ok@REDACTED>
> wrote:
> >
> > On 5/10/2011, at 5:37 AM, Peer Stritzinger wrote:
> > [how about a linear algebra library built on top of binaries, not
> entirely
> > unlike NumPy]
> >
> > Hasn't something like this already been done?  I'm sure I remember
> reading
> > about it.
>
> I have to admit I was not aware of this. OTOH it seems not to be
> available, can't find anything except the paper and EEP7 which is the
> foreign function interface to the external number crunching libraries
> they invented.
>
> However I was thinking along different lines, the approach of "HPTC
> with Erlang" (and also NumPy) is to slap on the big chunk of proven
> numerical routines as some external library.  Which is the way to go
> if you want to do serious number crunching, since its quite hard to
> develop trusted and efficient numerical routines.
>
> The price you have to pay for the slapped on heavyweight library is
> that these usually don't scale up to the number of processes Erlang
> can handle.  Therefore the need of the impedance adaption I mentioned.
> Keeping a pool of numerical processes to keep the cores busy but not
> too many of them that the OS is upset. Having work queues that adapt
> these to the 20k processes. BTW @John: this would be one solution for
> your problem.
>
> What I was suggesting is a more integrated and lightweight way to make
> some number crunching available.  The suggested n-dim matrix type
> (e.g. a record containing the metadata and a binary for the data)
> combined with some NIFs on these that speed up the parts where Erlang
> is not so fast.  Keeping in mind not to do too much work in the NIFs
> at one time not to block the scheduler.
>
> This is for the use cases where there is some numerical stuff needed
> but having real time responsiveness and Erlang process counts in mind.
>  The use case I have e.g. is some neuronal networks stuff combined
> with a lot of symbolic computing to prepare the input.  And its
> embarrassingly parallel and needs only some simple vector times matrix
> and n-dim array slicing.
>
> For real heavy numerical stuff I think the best way is to do this in
> the systems are built for this and interface them somehow to erlang
> with ports or sockets.  Or try to get the code released from the HPTC
> paper Jachym mentioned.
>
> For interfacing with BLAS and their ilk some more native Erlang
> numerical capabilities would also be nice to have.  Since they also
> use a kind of binary buffer with some metadata approach it would not
> be too hard to interface efficiently.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20111005/da92220a/attachment.htm>