[erlang-questions] Keeping massive concurrency when interfacing with C

[John Smith]

Thanks for the reply, Kresten!

I definitely would not be doing any disk I/O in the C code. It would
be intense number crunching, so it would be CPU (and perhaps memory)
bound. Everything I've read states Erlang is not good at number
brunching (Cesarini mentions this in his "Erlang Programming" book) so
I'm considering writing the code to do that in C.

If I call a NIF, only the particular scheduler that manages that
Erlang process would be blocked and no other scheduler, right? So for
example, if I have a CPU with eight cores, and an Erlang scheduler
thread is running on each core, and say the third scheduler is
executing an Erlang process that calls a NIF (and so blocks), only
that scheduler would be blocked until the NIF finishes executing,
correct?

I'm debating which solution would be better. Erlang would be slower at
number crunching, but is extremely efficient at managing concurrent
executing processes, meaning each would gradually make progress every
X units of time since they'll all get a turn to execute. But I wonder
if having a single process execute NIF code until it finishes (and so
all the processes managed by a single scheduler execute serially)
would be faster than implementing it all in Erlang and having
processes execute concurrently within a single scheduler (albeit the
code would be slower to execute). There would be less overhead of
Erlang process context switching (although admittedly that isn't much
to begin with) and the C code would be faster at number crunching. I
suppose there's only one way to find out! :)

I was also thinking about writing the number crunching code in some
other language than C, such as OCaml. OCaml has a reputation for being
as fast as C, yet not nearly as low-level. Maybe that would be a good
fit with Erlang.

Example benchmarks:
http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=hipe&lang2=gpp

http://shootout.alioth.debian.org/u32q/benchmark.php?test=all&lang=ocaml

The Erlang benchmark was using HiPE as well.

Thanks for the suggestion!