[erlang-questions] Wrapping C libraries in pure Erlang

[Denis Bilenko]

Romain Lenglet wrote:
> There exists a separate OS thread pool for exclusive use by linked-in
> drivers, but your drivers must explicitly programmed to use them, and
> they have severe drawbacks, e.g., you cannot communicate with Erlang
> (i.e., send a message to an Erlang process) from within such an I/O thread.
Do you mean that it cannot be done in a thread-safe way?
>From doc on erl_driver:
driver_output_term:
  Note that this function is not thread-safe,
  not even when the emulator with SMP support is used.
driver_send_term:
  This function is only thread-safe when the emulator
  with SMP support is used.
(http://www.erlang.org/doc/doc-5.5.4/erts-5.5.4/doc/html/erl_driver.html)

I assume this means that these functions must be called only from
driver_entry callbacks, never from worker threads.
EDTK uses ready_async callback to send term (with driver_output_term()).

> To understand why this cannot be trivial using the current linked-in
> driver mechanism implementation, please read about EDTK and Dryverl:
> http://www.erlang.se/workshop/2002/Fritchie.pdf
> http://www.csg.is.titech.ac.jp/paper/lenglet2006dryverl.pdf

Thanks, these are useful.
Other docs in EDTK are also very readable.
(found in http://www.snookles.com/erlang/edtk/edtk-1.5-candidate-2.tar.gz)

> If you use anything else than BIFs, then one important drawback is that
> you *must* pass all data as serialized terms between Erlang and C code.
> (Except that in some cases, you can send binaries to a linked-in drivers
> by reference, but that is quite tricky, cf. the paper on Dryverl)
> The linked-in drivers mechanism was not designed to interface to
> arbitrary C code. It was designed to implement I/O drivers. Period.
> Using this mechanism for anything else is possible, but is painful,
> which motivated tools like EDTK and Dryverl.
> If you really want to "manuipulate C data structures from Erlang", then
> you must use BIFs, which are undocumented and unsupported, i.e., your
> BIFs' code will be dependent on one specific version of the emulator,
> and you will have to distribute your own modified emulator to users.
> Impractical.
OK, then BIFs are out.
Serialization is a slowdown, but not a fatal problem and is
faced by every driver writer.

> Your hypothetical 'cee' library cannot be implemented in Erlang, since
> we have no control over the scheduler and the creation / allocation of
> scheduling OS threads from Erlang code, and any time Erlang code calls a
> BIF or interact with a linked-in driver, this is done in a scheduler's
> OS thread. If you have many simultaneous calls to C code from Erlang,
> which probability is increased when your calls to C code are blocking or
> take a long time, then you have less OS threads available from the pool
> to execute Erlang code. This could even stall or deadlock your application.

> If your library were in C, that would be OK, and the existing APIs are
> probably sufficient, but you have to do a lot to circumvent the limits
> of the asynchronous I/O threads, as I mentioned below:
Perhaps I wasn't clear -- I didn't expect cee library itself to be implemented
in pure Erlang. Its implementation will definitely involve some C code, in
form of a driver (or port, or c-node). Once it's done user gets pure-erlang
interface for calling functions in arbitrary dll/so.

> That is the kind of problems that EDTK and Dryverl try to solve. No need
> for yet another 'cee' library, IMO.

Let compare driver generated by EDTK with the one I'd like to have:

EDTK generated driver, when called by port_command, receives
  * function's id
  * packed arguments
then performs switch on received function id, thus selecting
  1) pointer to (statically linked) function
  2) procedure for decoding arguments (and encoding result)
  3) async parameter - whether to use dryver_async or call function directly
then it performs call either right away or through dryver_async,
constructs term and sends it to the port.
Also it does:
  a) check result against expected value, and obtain additional error info
     example: if (!res) error = errno; // error goes into output term
  b) maintain list of resources to free in case of process/port crash (valmaps)
  c) arbitrary hacks -- snippets of code in predefined places inserted as
     is from XML definition.

cee library driver would receive
  * function's name and a handle to a dll,
  * arguments with necessary type information
    example: [{double, 3.14}, {char_p, <<"Hello">>, inout}]
  * type of a return value
  * async parameter - whether to use driver_async or call function directly.
then it would pack arguments in libffi structure and call the function
in the dll.

So it is possible to do and not too far away from what EDTK already does.

What about a, b and c?
Well,
a) would be definitely necessary, since errno won't hang around too long.
    This could be done with predefined options:
    read errno when result is NULL / not NULL / negative / is not X ...
    Not as flexible as EDTK and Dryverl, but would suffice for many cases.
b) valmap is a tricky thing. Why implement it in C?
    In Erlang we can limit scope of a resource using try/after,
    or by monitoring (or linking to) process.
c) cannot be done.
    If some additional stuff need to be done at driver level, it either
     1. implemented for everybody and made available via option to port_call
     2. not available
    Again, not as flexible, but simpler.

Loaded C libraries won't be a pleasure to use as is.
Additional layer of Erlang code will be required to handle
resources and do other stuff to make library behave Erlangish.
But it seems much more natural than code generation.

So it can hardly be called 'yet another', it is different
from EDTK/Dryverl. (no XML, no generated drivers, simple to
use for simple cases)

> Particularly, Chris Newcombe seems to have done a great job dealing
> with heavy multithreading in linked-in drivers using EDTK.

Right, it seems that dryver_async is not good enough for all purposes.
Chris Newcombe in edtk-1.5/README-cnewcom wrote:
> Of course it is critical that the Erlang VM's scheduler threads are
> never blocked by Berkley DB. The standard, supported way to achieve this is
> using the Erlang 'async thread pool' ('erl +A' and the driver_async()
> API).  Unfortunately that mechanism is not flexible enough for the
> Berkeley DB driver (and using it would risk interfering with other
> important drivers like efile_drv). So EDTK now implements private
> threadpools, and multiplexes commands across those pools.  Each driver
> instance (port) has it's own set of threads, and the pools are
> resizeable at runtime.
This can be done for the cee library too.

Denis.