[erlang-questions] RFC: Erlang FFI

[Alceste Scalas]

Scott Lystig Fritchie <fritchie@REDACTED> wrote:
> Most of the Erlang<->C/C++ efforts have been SWIG-like efforts to
> present as smooth an interface to the non-C side as possible.  In
> Erlang, IMHO(*), such integration has been even more important.  At
> least half of Erlang's advantage is its ability to survive faults(*),
> and creating arbitrary interfaces into unknown shared libraries is a
> good way to invalidate much/all of those survival mechanisms.
> [...]
> Such an interface to libffi makes it possible to create Erlang
> interfaces to *any* shared library, without recompiling or relinking
> or other steps.  To create the same "smooth" interface, the glue ends
> up being on the Erlang side of the divide.  That philosophy is
> opposite from most FFI efforts in the Erlang world.
>
> There are 2 main FFI camps, it seems to me(*):
>
>     1. I don't care about fault tolerance, I *must* access to function
>        F in library L, and to hell with the consequences of buggy
>        code.
>
>     2. I really care about fault tolerance ... because if I didn't, I
>        wouldn't bother with Erlang at all.

I would like to point out something that I forgot to emphasize in my
previous email: the expected uses of the FFI BIFs I'm proposing.

Just like ddll:load/3, erlang:port_control/3 and friends, the FFI BIFs
(ffi:call/3, ddll:load_library/3, ffi:call/2, ffi:ecall/3) are *not*
intended to be used by "standard" Erlang developers.  They are
actually targeted at two kinds of, let's say, Erlang power developers:

     1. library bindings developers;

     2. developers that need to interface Erlang code with
        C/Fortran/Assembly/whatever (e.g. for performance reasons, thus
        making even the call speed a relevant factor).

This is directly related to your fault-tolerance and stability
worries.  The FFI BIFs could be used to crash the Erlang VM more or
less like a linked-in driver could [1].  But unaware developers that
just want to use libfoo from Erlang are usually *not* directly exposed
to the gore details of Erlang drivers: they don't see
erlang:port_control/3 (or ffi:call/3), but a documented Erlang libfoo
API that takes care of validating the parameters and performing the
underlying machinery.  If a developer decides to bypass this API layer
and access libfoo by fiddling with erlang:port_control/3 (or
ffi:call/3), then he/she is definitely on his/her own.

The main difference between linked-in drivers and FFI BIFs, IMHO, is
that ffi:call/3 and friends make it a lot easier to develop library
bindings and to interface external code to Erlang, both for
quick'n'dirty hacks and well-refined products.  The FFI approach is
also more powerful, expecially with the proposed ffi:ecall/3
BIF [2].

I agree when you write that, when safety and fault-tolerance are
fundamental and "untrusted" code could not be allowed, the ideal
solution would be to run the untrusted FFI code in a separate process.
But it may require quite some work.  From this point of view, I agree
with Denis: instead of developing more-or-less transparent pipe driver
with complicated serialization (as pointed out by Vlad), one may run
one or more Erlang nodes with the unsafe stuff, and issue remote calls
through a simplified API layer.  This could be the cheaper and
possibly more flexible way to obtain both FFI and greater
fault-tolerance (at the price of speed, of course).  And this approach
could even be used with "untrusted" linked-in drivers.

Regards,

alceste

Notes:

[1] Well, one may program an erlang:port_control/3 C callback that
     reacts gracefully to every possible kind of garbage being received
     as binary data.  But the same defensive programming could be done
     for C functions expected to be called through ffi:call/3.  Being
     able to call arbitrary C functions through FFI BIFs doesn't
     prevent a library binding developer to create and document a set
     of "safe" C functions that could be called from Erlang, with the
     warning that other calls may have unexpected consequences.

[2] Let's see an "extreme" scenario: something like ffi:ecall/3
     with some glue code based on libffi closures would allow (at least
     in theory) to pass an Erlang fun() to the C side to be used as a
     callback, even from code that is completely Erlang-unaware (as
     an example: write a differential equation in Erlang and let the
     GNU Scientific Library integrate it).  I didn't find a way (yet)
     to apply/3 an Erlang fun() from C code, but I didn't do much
     investigation in the Erlang VM internals.  The possibility to
     call C-from-Erlang and Erlang-from-C, however, would increase
     the Erlang language interoperability to the levels of Python,
     Java, etc.
-- 
Alceste Scalas <alceste@REDACTED>
CRS4 - http://www.crs4.it/

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.