[erlang-questions] Wrapping C libraries in pure Erlang

[Romain Lenglet]

Denis Bilenko wrote:
> Serge Aleynikov wrote:
> 
>> Part of the issue is that this approach would make it possible for a
>> function call inside a 3rd party library to block for a while.  Since
>> Erlang's concurrency is dependent on the fact that function calls are
>> very short, and the emulator uses reduction-based counting for giving
>> CPU slices to each light-weight process a blocking call would
>> significantly inhibit concurrency.
> 
> For that case, imaginary library 'cee' has function 'cast' that instructs
> driver to call function asynchronously, as you have mentioned below.

That does not work, because while an emulator scheduler's OS thread is 
calling C code, it cannot execute Erlang code. Even if your process uses 
cee or cast, the C code will still be executed using one of the 
scheduler's OS threads that are also used to execute Erlang code. There 
is no separate OS thread pool for Erlang and C code. Therefore, this 
doesn't solve the problem.

There exists a separate OS thread pool for exclusive use by linked-in 
drivers, but your drivers must explicitly programmed to use them, and 
they have severe drawbacks, e.g., you cannot communicate with Erlang 
(i.e., send a message to an Erlang process) from within such an I/O thread.

That is what Serge Aleynikov mentioned just below:

>> Drivers solve this problem by allowing asynchronous invocation of
>> blocking functions in the context of threads different from the
>> emulator(s) (*).  In case of a driver or a port, once you are on the
>> C-side of coding interface, you can make calls to any 3rd party library
>> directly without needing FFI (except for cases of functions with
>> variable arguments), which may bring you to the question of why bother
>> with such an Erlang library if you already have ports and drivers that
>> can be written in C or even ei interface for writing C-nodes that can
>> make any C-calls safely with respect to the emulator?
> 
> Because, if I had such a library I would not have to bother
> with C, ei, driver interface.
> The point is, why manipulate Erlang data structures from C instead of
> manipulating C data structures from Erlang.
> Latter approach has to be more productive for (Erlang) programmer.

To understand why this cannot be trivial using the current linked-in 
driver mechanism implementation, please read about EDTK and Dryverl:
http://www.erlang.se/workshop/2002/Fritchie.pdf
http://www.csg.is.titech.ac.jp/paper/lenglet2006dryverl.pdf

There has also been IG around for some time, but it does not support 
linked-in drivers:
http://www.bluetail.com/tobbe/ig/doc.new/

If you use anything else than BIFs, then one important drawback is that 
you *must* pass all data as serialized terms between Erlang and C code. 
(Except that in some cases, you can send binaries to a linked-in drivers 
by reference, but that is quite tricky, cf. the paper on Dryverl)
The linked-in drivers mechanism was not designed to interface to 
arbitrary C code. It was designed to implement I/O drivers. Period. 
Using this mechanism for anything else is possible, but is painful, 
which motivated tools like EDTK and Dryverl.
If you really want to "manuipulate C data structures from Erlang", then 
you must use BIFs, which are undocumented and unsupported, i.e., your 
BIFs' code will be dependent on one specific version of the emulator, 
and you will have to distribute your own modified emulator to users. 
Impractical.

If it were that easy to do, please believe that such an automatic 
integration of C and Erlang would have been done a long time ago. (^_^)

Therefore, I agree with Mats and Serge: if you need simple concurrency 
and flexibility, use C ports. Linked-in drivers potentially give better 
performance, but may not be worth the effort for your problems.

--
Romain Lenglet