[erlang-questions] Keeping massive concurrency when interfacing with C

[Kresten Krab Thorup]

Hi John.

You'd most likely want to implement anything that blocks in Erlang.  Typically, Erlang runs as many threads as there are cores (can be controlled with the +S erl argument). 

The question is, what is it that you want to do in that C code?  Because the hardware is not able to do more things concurrently than it has cores..

If you want to do I/O in the c code, then you're probably better off writing it in Erlang, or use one of the existing linked in drivers designed for doing just that.

NIFs are best suited to do stuff that requires heavy computation, or something that needs access to a shared data structure, or leveraging some existing code.  ...but NIFs should not run for too long since they'll hold up all the other Erlang processes on that scheduler, and NIFs should generally not do I/O.

So if you have a large body of existing c code which does I/O, then there's no easy way to combine that with the benefits of erlang's inexpensive processes.

Sent from my iPad

On Oct 2, 2011, at 23:10, "John Smith" <emailregaccount@REDACTED> wrote:

> Hi everyone,
> 
> From my understanding, there are four main ways to interface Erlang
> with C:
> 
> * C Node
> * Port
> * Linked-In Driver
> * NIF (Native Implemented Function)
> 
> My problem is if I have, for example, spawned 20,000 Erlang processes
> and I want them all to execute concurrently but they need to call C
> code, how can I have that C code run concurrently without having to
> spawn 20,000 threads in C (which would probably crash the OS) or using
> obscene amounts of memory?
> 
> I've been reading over the examples of a C node, and it seems if
> 20,000 processes all send a message to the C node, the node will
> process them one-by-one and not concurrently, so it becomes a
> serialized bottleneck. Spawning 20,000 C nodes on a single machine
> isn't feasible, because of the amount of memory that would require.
> 
> A port suffers from the same problem, since the Erlang processes would
> be communicating with a single external program, and again, I can't
> create 20,000 instances of that program.
> 
> Reading the documentation for a linked-in driver, it says:
> http://www.erlang.org/doc/tutorial/c_portdriver.html
> 
> "Just as with a port program, the port communicates with a Erlang
> process. All communication goes through one Erlang process that is the
> connected process of the port driver. Terminating this process closes
> the port driver."
> 
> But on the driver documentation page:
> http://www.erlang.org/doc/man/erl_driver.html
> 
> "A driver is a library with a set of function that the emulator calls,
> in response to Erlang functions and message sending. There may be
> multiple instances of a driver, each instance is connected to an
> Erlang port. Every port has a port owner process. Communication with
> the port is normally done through the port owner process."
> 
> So this also seems to have the same problem as C nodes and ports,
> since in order to maintain concurrency I would need 20,000 instances
> of the same driver.
> 
> Finally, we have NIFs. These have potential, but when I read the
> documentation:
> http://www.erlang.org/doc/man/erl_nif.html
> 
> "Avoid doing lengthy work in NIF calls as that may degrade the
> responsiveness of the VM. NIFs are called directly by the same
> scheduler thread that executed the calling Erlang code. The calling
> scheduler will thus be blocked from doing any other work until the NIF
> returns."
> 
> So if one Erlang process calls a NIF, does this mean the other 19,999
> processes are blocked until the NIF returns (or the subset of
> processes a scheduler manages)? If so, this won't work either.
> 
> Does anyone have a solution to this that still allows you to use C
> (I'm using C for the parts that are intensive number crunching)? Or
> will I have to implement everything in Erlang?
> 
> Thanks!
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions