[erlang-questions] RFC: Erlang FFI

[Alceste Scalas]

Hello,

following the rules of the EEP workflow [1], I'd like to know your
opinions about extending Erlang with a Foreign Function Interface
(FFI).

I've got an implementation based on GCC's FFI library [2], that is
mostly complete [3] and tested on GNU/Linux on AMD64.  The first
part of the following description refers to this implementation,
while the rest of the email contains ideas for possible
enhancements.

Depending on your feedback, I'll provide the patch for my current
Erlang FFI implementation, extend it and/or write a proper Erlang
Enhacement Proposal.

A few notes about the GCC FFI library: it is released under a very
permissive license, it supports a whole lot of architectures,
and it is available as a separate package (for Debian/Ubuntu
users: apt-get install libffi4 libffi4-dev ).  If there's any
Pythonista out there: it's the library on which Python ctype
module is built.

Regards,

alceste
- - - -
1. The basic API
================

The Erlang FFI would consist in (at least) one BIF:

    ffi:call(Handler, Call::call_tuple(),
             Types::types_tuple()) -> return_type()

    call_tuple() -> {Fn, Arg1, ...}

    types_typle() -> {RetType, TypeArg1, ...}

"Handler" is an handler for the dynamic library containing the
function to be called (in the current implementation, it is a port
obtained by loading an Erlang linked-in driver).  "Fn" is an atom or
an io_list with the function name.  RetType, TypeArg1 etc. are atoms
representing the C type of the function and its arguments.  The
return_type() of ffi:call/3 is thus determined by RetType.

A simple example:

    Hello = list_to_binary(["Hello, world!", 0]),
    RetVal = ffi:call(Handler, {puts, Hello}, {sint, pointer}).

It causes the C function puts() to output the null-terminated string
contained in the "Hello" binary.  Its C return value (a signed
integer) is converted into an Erlang integer (small or big depending
on its value).

The supported C types are represented with the following atoms:

    uint      uint8   float
    sint      sint8   double
    ushort    uint16  longdouble
    sshort    sint16  pointer
    ulong     uint32  void
    slong     sint32
    uchar     uint64
    schar     sint64

The Erlang terms passed as ffi:call/3 arguments are converted to the
proper C types.  Binaries are passed as pointers.  If the call returns
a pointer, it is converted into an Erlang integer (small or big).


2. Enhancement: the library handler
===================================

In the current implementation, ffi:call/3 expects "Handler" to be a
(possibly registered) port.  It is not optimal, expecially because in
order to call functions from the "foo" library it is necessary to
create a C file with a void ErlDrvEntry structure, compile and link it
against libfoo itself, and finally load it inside the Erlang VM (by
using ddll:load_driver/2 and friends).

A cleaner interface for loading an arbitrary DLL would be handy,
e.g. something like:

    Handler = ddll:load_library(Path, Name, OptionList).

Where "Handler" would be an opaque reference to the proper entry in
the VM's internal list of loaded drivers.  Maybe it could be just a
port, that ddll:load_library/3 could automatically associate with a
void ErlDrvEntry structure: it would be very easy to implement, and
quite consistent with the rest of the Erlang port machinery.


3. Enhancement: FFI call optimizations
======================================

The OptionList parameter of ddll:load_library/3 could be used for a
very effective optimization: preloading some symbols and precompiling
their FFI call structure.  One example:

    Handler = ddll:load_library("/lib", c,
                                {preload,
                                 [{puts, {sint, pointer}},
                                  {putchar, {sint, sint}},
                                  {malloc, {pointer, uint}},
                                  {free, {void, pointer}}]}).

This preloading mechanism would be used by another BIF, ffi:call/2:

    MemPtr = ffi:call(Handler, {3, 1024})
    ffi:call(Handler, {4, MemPtr})

Where "3" and "4" are the positions of the functions in the "preload"
list.  This call would be definitely faster than the one performed by
ffi:call/3.


4. Enhancement: BIF-like external functions
===========================================

The FFI support opens the possibility to pass Erlang terms directly to
the C world.  For example, with a ffi:ecall/3 BIF like:

    ffi:ecall(Handler, do_something, {Arg1, ...})

The terms given as arguments may be passed directly to the C side,
eventually along with the current Process* (thus making external
functions something like add-on BIFs).  Of course, in order to make it
work, it would be necessary to compile the C source against a matching
version of the OTP header files.

Since it's extremely powerful and gives developers enough rope to
shoot themselves in their foots, the ffi:ecall/3 BIF could be a
not-so-visible-and-documented resource (i.e. something like the
powerful HiPE BIFs).


Notes and references
====================

[1] http://www.erlang.org/eeps/eep-0001.html
[2] http://gcc.gnu.org/viewcvs/trunk/libffi/
[3] The only thing left to be done is converting C signed int and
    signed long variables to/from Erlang signed big integers.

-- 
Alceste Scalas <alceste@REDACTED>
CRS4 - http://www.crs4.it/