[erlang-questions] RFC: Erlang FFI

[Scott Lystig Fritchie]

>>>>> "as" == Alceste Scalas <alceste@REDACTED> writes:

as> A few notes about the GCC FFI library: it is released under a very
as> permissive license, it supports a whole lot of architectures, and
as> it is available as a separate package (for Debian/Ubuntu users:
as> apt-get install libffi4 libffi4-dev ).  If there's any Pythonista
as> out there: it's the library on which Python ctype module is built.

Alceste: This is good news.  A couple months ago, I'd stumbled across
libffi and Python's interface to it.  I thought, "That'd be nifty, in
a very-easy-to-shoot-yourself-in-the-foot kind of way."  I also had
absolutely no time to pursue it, so I'm glad that someone else is.

Most of the Erlang<->C/C++ efforts have been SWIG-like efforts to
present as smooth an interface to the non-C side as possible.  In
Erlang, IMHO(*), such integration has been even more important.  At
least half of Erlang's advantage is its ability to survive faults(*),
and creating arbitrary interfaces into unknown shared libraries is a
good way to invalidate much/all of those survival mechanisms.

However, such control is difficult to manage.  And more than a bit
chafing, mentally.  And so there still aren't any easy-to-use FFI
thingies.

Such an interface to libffi makes it possible to create Erlang
interfaces to *any* shared library, without recompiling or relinking
or other steps.  To create the same "smooth" interface, the glue ends
up being on the Erlang side of the divide.  That philosophy is
opposite from most FFI efforts in the Erlang world.

There are 2 main FFI camps, it seems to me(*):

    1. I don't care about fault tolerance, I *must* access to function
       F in library L, and to hell with the consequences of buggy
       code.

    2. I really care about fault tolerance ... because if I didn't, I
       wouldn't bother with Erlang at all.

I used to be in the first camp.  As I've developed more apps, and seen
Erlang generic servers and supervisor trees and other Erlang/OTP
wizardry keep my emergency 24-hour support pager completely silent,
I'm quite firmly in the second camp.  8-)

A raw interface with libffi would fall into the first camp.  My
suggestion for you is to think long and hard about also supporting the
2nd camp, as much as possible.  Because there are plenty of us that
would love to use special library L in an Erlang app in a *production*
environment, but won't run the risk of using L without as many
Erlang'ish safeguards as possible.

For example, being able to choose whether to use the FFI via dlopen()
1. inside the Erlang VM, or 2. in a separate OS process (communicating
via pipe, socket, whatever).  Preferably using the same interface.

Personally, I'm a big fan of the VM's efforts to clean up ports, file
handles, sockets, etc. whenever a process crashes.  For long term
survivability, it is extremely useful if resources allocated by
library L can also be cleaned up/freed/closed/whatever when the
"owner" process crashes.  Such bookkeeping in Erlang is trickier to
avoid leaks than if it's done inside a driver/BIF on the C side, but
perhaps it's as feasible (with some creative thinking)?

Prior FFI efforts have tried to make it difficult to do suicidal stuff
like:

    BadBufferAddress = 88044,		% Clobber something else, if you wish!
    Fmt = list_to_binary(["Hello, world!", 0]),
    ffi:call(Handler, {sprintf, BadBufferAddress, Fmt}, {sint, pointer}),

... by first requiring a memory allocation for the buffer, and somehow
tagging that pointer so that a random integer (or some other pointer
type) can't be accidentally used by sprintf().  Hm, for a first libffi
effort, such safety probably isn't worth it.  Camp #1 people won't
want to wait for the safety feature, and Camp #2 people will be either
a) patient and will wait, or b) implement their own safety net.

For either camp, releasing an implementation of a libffi interface
would be better sooner than waiting for zillions of features(**).

-Scott

(*) An opinion probably not shared by everybody.  :-)

(**) Less disagreement, I think?