[erlang-questions] Wrapping C libraries in pure Erlang

[Romain Lenglet]

Romain Lenglet wrote:
> Denis Bilenko wrote:
>> Serge Aleynikov wrote:
>>
>>> Part of the issue is that this approach would make it possible for a
>>> function call inside a 3rd party library to block for a while.  Since
>>> Erlang's concurrency is dependent on the fact that function calls are
>>> very short, and the emulator uses reduction-based counting for giving
>>> CPU slices to each light-weight process a blocking call would
>>> significantly inhibit concurrency.
>> For that case, imaginary library 'cee' has function 'cast' that instructs
>> driver to call function asynchronously, as you have mentioned below.
> 
> That does not work, because while an emulator scheduler's OS thread is 
> calling C code, it cannot execute Erlang code. Even if your process uses 
> cee or cast, the C code will still be executed using one of the 
> scheduler's OS threads that are also used to execute Erlang code. There 
> is no separate OS thread pool for Erlang and C code. Therefore, this 
> doesn't solve the problem.

OK... now that my cups of coffee start having effect, I can clarify my 
paragraph above.

Your hypothetical 'cee' library cannot be implemented in Erlang, since 
we have no control over the scheduler and the creation / allocation of 
scheduling OS threads from Erlang code, and any time Erlang code calls a 
BIF or interact with a linked-in driver, this is done in a scheduler's 
OS thread. If you have many simultaneous calls to C code from Erlang, 
which probability is increased when your calls to C code are blocking or 
take a long time, then you have less OS threads available from the pool 
to execute Erlang code. This could even stall or deadlock your application.

If your library were in C, that would be OK, and the existing APIs are 
probably sufficient, but you have to do a lot to circumvent the limits 
of the asynchronous I/O threads, as I mentioned below:

> There exists a separate OS thread pool for exclusive use by linked-in 
> drivers, but your drivers must explicitly programmed to use them, and 
> they have severe drawbacks, e.g., you cannot communicate with Erlang 
> (i.e., send a message to an Erlang process) from within such an I/O thread.

That is the kind of problems that EDTK and Dryverl try to solve. No need 
for yet another 'cee' library, IMO. Particularly, Chris Newcombe seems 
to have done a great job dealing with heavy multithreading in linked-in 
drivers using EDTK.

--
Romain Lenglet