This EEP describes a Foreign Function Interface (FFI) for Erlang/OTP,
that allows to easily perform direct calls of external C functions.
It introduces three new BIFs (ffi:raw_call/3,
erl_ddll:load_library/3 and ffi:raw_call/2) that accomplish the main
FFI tasks: loading generic C libraries, making external function calls
and performing automatic Erlang-to-C and C-to-Erlang type conversions.
It also introduces two auxiliary BIFs for converting C buffers/strings
into binaries (ffi:raw_buffer_to_binary/2 and
ffi:raw_cstring_to_binary/1), a new ffi Erlang module that
provides a higher-level API with stricter type checking, and some
utility macros. Finally, it extends erl_ddll:info/2 with FFI
information.
The current Erlang extension mechanisms can be divided in two main categories:
absolute stability at the price of speed (C nodes, pipe drivers);
more speed at the (potential) price of stability (linked-in drivers).
Linked-in drivers have thus become the standard way for creating library bindings when efficiency is an issue. In both cases, however, the Erlang driver interface implies the development of relevant amounts of glue code, mostly because the communication between Erlang and C always requires data parsing and (de)serialization. Several tools have been created in order to autogenerate (at least part of) that glue: from the (now unmaintained) IG driver generation tool to the newer Erlang Driver Toolkit (EDTK) and Dryverl.
But, even with the help of these tools, developing an Erlang driver is a difficult and time-consuming task (especially when interfacing external libraries with tens or hundreds of functions), and the glue code itself increases the possibility to introduce bugs — that, in the case of linked-in drivers, usually mean VM crashes. For all these reasons, the lack of libraries and the difficulty of interfacing them from other languages is one of the negative aspects that are usually associated with Erlang.
The same problems also arise when a developer needs to replace performance-critical portions of his/her Erlang code with optimized C functions. In this case, also the data serialization/deserialization overhead may be a significant issue.
An easier method for interfacing Erlang and C code could drastically extend the Erlang capabilities and open new usage scenarios.
This EEP proposes a Foreign Function Interface (FFI) extension that would allow to easily perform direct C function calls. This concept is implemented in almost every language, with two main (non-exclusive) approaches:
automatic type conversions between the host and the foreign language (examples: Python, Haskell);
documented C interface for handling host language types from the foreign language (examples: Java, Python (API)).
This EEP follows the first approach, but (when possible) also reuses
part of the existing C Driver API (and, thus, allows to manage
ErlDrvBinary and ErlIOVec pointers in the external C
functions).
The FFI has been designed not to require language changes or introduce incompatibilities.
The BIFs and functions proposed in this EEP don’t give any access to the Erlang VM internals — but the called C functions could leak memory and/or cause the Erlang VM to crash. The FFI is, thus, not intended for “casual” Erlang developers: this is a tool designed for library bindings developers (that should take care of hiding FFI calls from final users), and advanced programmers looking for an easy (and efficient) way to call C code from Erlang.
In order to call a C function, the FFI needs a port opened towards the required C code. Thus, with the current driver loading mechanism, a developer would be required to:
create a C file with a void ErlDrvEntry structure and driver
init function;
compile it and possibly link it against the required C libraries, thus obtaining a void Erlang driver;
load the driver in the Erlang VM, by using erl_ddll:load/2.
In order to simplify this procedure, this EEP proposes the
erl_ddll:load_library/3 function, that allows to load a generic
library in the Erlang VM — even if it lacks the structure of an
Erlang linked-in driver.
erl_ddll:load_library/3 also offers an option to preload a list of C
function symbols and signatures, thus precompiling the internal
structures needed for performing dynamic function calls. Information
about preloaded data can be retrieved with erl_ddll:info/2.
Once a library or driver has been loaded, erlang:open_port/2 or
erlang:open_port/1 could be used to get a port for the FFI
functions, and perform calls either through the low-level or the
high-level APIs.
The low-level FFI methods are denoted by the raw_ prefix. The
main function is the ffi:raw_call/3 BIF, that performs a direct C
function call through an open port. It converts C types to/from
Erlang types.
When taken alone, ffi:raw_call/3 has got a major drawback: it introduces
great call overhead, due to the C symbol lookup and the dynamic
construction of the function call.
In order to exploit preloading option of erl_ddll:load_library/3, the
ffi:raw_call/2 BIF is introduced: it avoids symbol lookup and call
structure compilation, thus guaranteeing a lower call overhead than
ffi:raw_call/3.
Furthermore, the low-level interface provides two BIFs for creating an
Erlang binary from a C pointer (possibly returned by a FFI call).
These BIFs are ffi:raw_buffer_to_binary/2 and
ffi:raw_cstring_to_binary/1.
The high-level interface is built upon the low-level one. It
introduces the concept of type-tagged values: any value passed to or
returned from FFI calls has the form of a {Type, Value} tuple. This
allows to:
increase the readability of FFI calls;
make the C calls safer: the consistency of tagged values is checked
before the values themselves are passed to the low-level API.
Furthermore, the preload information given to
erl_ddll:load_library/3 is used (when available) to ensure that the
tagged values actually match the function signature;
simulate the static typing of C code, thus requiring proper and explicit “casts” when a tagged value needs to be converted to another type.
These checks are performed by ffi:call/3, ffi:buffer_to_binary/2
and ffi:cstring_to_binary/1 (the type-tagged equivalents of the
low-level BIFs). Type-tagged values can also be checked with
ffi:check/1. Furthermore, the allowed minimum and maximum value of
each FFI type can be examined with ffi:min/1 and ffi:max/1.
The FFI defines a series of utility macros in the ffi_hardcodes.hrl
header file, that could be used for binary matching of C buffers and
structures.
c_func_name() #
c_func_name() = atom() | string()
Name of a C function.
type_tag() #
type_tag() = atom()
Valid FFI type atom. For the list of allowed values, see the Appendix.
tagged_value() #
tagged_value() = tuple(type_tag(), term())
Type-tagged value used for FFI calls.
tagged_func_name() #
tagged_func_name() = tuple(type_tag(), c_func_name())
C function name with return type.
func_index() #
func_index() = integer()
Function position on the list of preloads given to
erl_ddll:load_library/3.
tagged_func_index() #
tagged_func_index() = tuple(type_tag(), func_index())
C function index with return type.
signature() #
signature() = tuple(type_tag(), ...)
Signature of a C function: return type followed by arguments types (if any).
erl_ddll:load_library/3 #
erl_ddll:load_library(Path, Name,
OptionsList) -> ok | {error, ErrorDesc}
Types:
Path = Name = string() | atom()
OptionList = [Option]
Option = tuple(preload, [Preload])
Preload = tuple(c_func_name(), signature())
Load a generic shared library.
If an ErlDrvEntry structure and a driver init function are found when
loading the library, this BIF will behave like erl_ddll:load/2. The
function parameters are also the same of erl_ddll:load/2, with the
following addition:
OptionList is a list of options for library/driver loading. The supported options are:
{preload, PreloadList}
Preload the given list of functions, and prepare their call structures. Each PreloadList element is a tuple in the form:
tuple(c_func_name(), signature())
i.e. the function name followed by its return and arguments types.
The function return values are the same of erl_ddll:load/2.
Once a library has been loaded, it is possible to use
erlang:open_port/2 to get a port. That port could always be used
with ffi:call/3, ffi:raw_call/3 or ffi:raw_call/2. However,
if the loaded library does not contain a proper ErlDrvEntry
structure and a driver init function, the port will not be usable
with erlang:port_command/2, erlang:port_control/3 etc.
The following example loads the C standard library and preloads some functions:
ok = erl_ddll:load_library("/lib", libc,
[{preload,
[{puts, {sint, nonnull}},
{putchar, {sint, sint}},
{malloc, {nonnull, size_t}},
{free, {void, nonnull}}]}]).
erl_ddll:load_library/2 #
erl_ddll:load_library(Path, Name)
Utility function that calls erl_ddll:load_library/3 with an empty
OptionsList.
erlang:open_port/1 #
erlang:open_port(Library)
Types:
Library = string() | atom()Open a port towards the specified shared library, possibly loaded with
erl_ddll:load_library/3. Calling this function is equivalent to:
erlang:open_port({spawn, Library}, [binary])
erl_ddll:info/2 #
This EEP proposes a new parameter for the erl_ddll:info/2 BIF: the
‘preloads’ atom. It allows to retrieve information about FFI preloads
for the given library.
The preload information is a list of proplists, one for each preloaded function. Each proplist, in turn, has the following format:
[ { index, integer() }, % Position in the preload list
{ name, string() }, % Function name
{ address, integer() }, % Function address
{ signature, signature() } ] % Function signature
This information would be made available also through erl_ddll:info/0
and erl_ddll:info/1.
ffi:raw_call/3 #
ffi:raw_call(Port, CallArgs, Signature) -> term()
Types:
Port = port()
CallArgs = tuple(c_func_name(), Arg1, ...)
Arg1, ... = term()
Signature = signature()
Call the specified C function.
This BIF accepts the following parameters:
Port
A port opened towards the required driver/library.
CallArgs
A tuple with the function name (atom or string) followed by its arguments (if any).
Signature
Function signature.
This BIF returns the return value of the C function being called (or ‘void’ if the return type is void). It automatically converts Erlang terms to/from C values. The supported C types and conversions are reported in the Appendix.
The following example calls the malloc() and free() functions from the
standard C library (it should work with any Erlang linked-in driver): ::
Pointer = ffi:raw_call(Port, {malloc, 1024}, {pointer, size_t}),
ok = ffi:raw_call(Port, {free, Pointer}, {void, pointer}).
WARNING: bugs and/or misuses of the external C functions can affect the Erlang VM, possibly making it crash. Use this BIF with extreme care.
ffi:raw_call/2 #
ffi:raw_call(Port, OptimizedCall) -> term()
Types:
Port = port()
OptimizedCall = {FuncIndex, Arg1, ...}
FuncIndex = func_index()
Arg1, ... = term()
Call a function preloaded with the ‘preload’ option of
erl_ddll:load_library/3.
This BIF accepts the following parameters:
Port
A port opened towards the required driver/library (that
must have been loaded with erl_ddll:load_library/3).
OptimizedCall
A tuple with the function index (i.e. its position in the preload list) followed by its arguments (if any).
This BIF returns the return value of the C function being called (or ‘void’ if the return type is void). It automatically converts Erlang terms to/from C values. The supported C types and conversions are reported in the Appendix.
The following example calls malloc() and free(), after they have been
preloaded with the code sample shown in erl_ddll:load_library/3:
Port = open_port({spawn, "libc"}, [binary]),
Pointer = ffi:raw_call(Port, {3, 1024}),
ffi:raw_call(Port, {4, Pointer})
WARNING: bugs and/or misuses of the external C functions can affect the Erlang VM, possibly making it crash. Use this BIF with extreme care.
ffi:raw_buffer_to_binary/2 #
ffi:raw_buffer_to_binary(Pointer, Size) -> binary()
Types:
Pointer = integer()
Size = integer()
Return a binary with a copy of Size bytes read from the given C pointer (represented by an integer, possibly returned by a FFI call).
WARNING: passing the wrong pointer to this BIF may cause the Erlang VM to crash. Use with extreme care.
ffi:raw_cstring_to_binary/1 #
ffi:raw_cstring_to_binary(CString) -> binary()
Types:
CString = integer()Return a binary with a copy of the given NULL-terminated C string (an integer representing a pointer, possibly returned by a FFI call). The binary will include the trailing 0.
WARNING: passing a wrong pointer to this BIF may cause the Erlang VM to crash. Use with extreme care.
ffi:call/3 #
call(Port, CFunc, Args) -> RetVal
Types:
Port = port()
CFunc = c_func_name() | func_index() | tagged_func_name() | tagged_func_index()
Args = [tagged_value()]
RetVal = tagged_value()
Call the C function CFunc with the given list of arguments, using
the port Port. If the function was preloaded with
ffi:load_library/3, all the type tags will be matched against the
preloaded signature before performing the call.
Return the return value of the C function, with the proper type tag.
Note: if CFunc is not of type tagged_func_name(), the C
function will be called if and only if it was preloaded with
erl_ddll:load_library/3 (it is required in order to determine its
return type).
As an example, the following malloc() calls are all valid and
equivalent when executed after the code sample shown in
erl_ddll:load_library/3:
%% Use function name, but require preloads for return type
{nonnull, Ptr1} = ffi:call(Port, "malloc", [{size_t, 1024}]),
{nonnull, Ptr2} = ffi:call(Port, malloc, [{size_t, 1024}]),
%% Use function index from preloads list
{nonnull, Ptr3} = ffi:call(Port, 3, [{size_t, 1024}]),
{nonnull, Ptr4} = ffi:call(Port, {nonnull, 3}, [{size_t, 1024}]),
%% These calls do not require any preload information
{nonnull, Ptr5} = ffi:call(Port, {nonnull, "malloc"}, [{size_t, 1024}]),
{nonnull, Ptr6} = ffi:call(Port, {nonnull, malloc}, [{size_t, 1024}]),
WARNING: bugs and/or misuses of the external C functions can affect the Erlang VM, possibly making it crash. Use this BIF with extreme care.
ffi:buffer_to_binary/2 #
ffi:buffer_to_binary(TaggedNonNull, Size) -> binary()
Types:
TaggedNonNull = tuple(nonnull, integer())
Size: integer()
Return a binary with a copy of Size bytes read from the given C
pointer.
WARNING: passing a wrong pointer to this function may cause the Erlang VM to crash. Use with extreme care.
ffi:cstring_to_binary/1 #
ffi:cstring_to_binary(TaggedCString) -> binary()
Types:
TaggedCString = tuple(cstring, integer())Return a binary with a copy of the given NULL-terminated C string.
WARNING: passing a wrong pointer to this function may cause the Erlang VM to crash. Use with extreme care.
ffi:sizeof/1 #
ffi:sizeof(TypeTag) -> integer()
Types:
TypeTag: type_tag()Return the size (in bytes) of the given FFI type, on the current platform.
ffi:check/1 #
ffi:check(TaggedValue) -> true | false
Types:
TaggedValue = tagged_value()Returns ‘true’ if the given type-tagged value is well-formed and consistent (i.e. it falls in the allowed range for its type, on the current platform). Otherwise, returns ‘false’.
ffi:min/1 #
ffi:min(TypeTag) -> integer()
Types:
TypeTag = type_tag()Return the minimum value allowed for the given FFI type, on the current platform.
ffi:max/1 #
ffi:max(TypeTag) -> integer()
Types:
TypeTag = type_tag()Return the maximum value allowed for the given FFI type, on the current platform.
ffi_hardcodes.hrl #
The ffi_hardcodes.hrl file is part of the Erlang ffi library. It
defines a set of macros for handling FFI types sizes, and for easy
binary matching on C buffers and structures:
FFI_HARDCODED_<TYPE>
An Erlang bit-syntax snippet (Size/TypeSpecifier) that could be used to match the given FFI type inside a binary (possibly obtained from a C buffer). For example, the following binary matching:
<<ULong:?FFI_HARDCODED_ULONG, _Rest/binary>> = Binary
on x86-64 will expand to:
<<ULong:64/native-unsigned-integer, _Rest/binary>> = Binary
FFI_HARDCODED_SIZEOF_<TYPE>
The type size in bytes
FFI_HARDCODED_<TYPE>_BITS
The type size in bits
As implied by their name, the ffi_hardcodes.hrl contents are
specific to the build platform, and when they are used, they will be
hard-coded in the resulting .beam files. Thus, these macros
should be avoided if a developer expects his/her FFI-based code to be
portable without recompilation. The recommended method for getting
FFI type sizes in a portable way is the ffi:sizeof/1 function.
When a library is loaded with erl_ddll:load_library/3, it may be
reloaded or unloaded just like any Erlang linked-in driver. If the
‘preload’ option is used, then two additional behaviors arise:
if erl_ddll:load_library/3 is called two or more times with the
same library, then the associated preload list must be rebuilt
according to the last call. If no ‘preload’ option is used, then
the last preloads (if any) must be kept intact;
if an erl_ddll:reload/2 is issued, then the last preloads must be
refreshed by performing a new symbol lookup in the loaded library.
If one or more symbols could not be found anymore, then they must be
disabled (and an error must raised when trying to use them with
ffi:raw_call/2).
ffi:call/3 and ffi:raw_call/3 may be used to call vararg C
functions, simply by providing the desired number of arguments.
In order to exploit the preloading optimizations, however, it is
necessary to use a different preload for each different function call
signature. For example, if a developer is going to call printf()
with different arguments, he/she will need to use a preloading list
like the following one:
ok = erl_ddll:load_library("/lib", libc,
[{preload,
[{printf, {sint, cstring}},
{printf, {sint, cstring, double}},
{printf, {sint, cstring, uint, sint}},
{printf, {sint, cstring, cstring}}]}]).
As reported in the Appendix, an Erlang binary can be passed to a C function as a ‘pointer’ value. In this case, the C function will receive a pointer to the first byte of binary data.
That pointer will be valid only until the C function returns. If the C side needs to access the pointer data later, then it should use the ‘binary’ FFI type (see next paragraph) or copy the data itself in a safe place.
As reported in the Appendix, when the ‘binary’ FFI type is used as
argument, the C function will also receive a binary (in the form of an
ErlDrvBinary pointer). Correspondingly, a C function with ‘binary’
FFI return type must return an ErlDrvBinary pointer. Furthermore,
an ‘erliovec’ argument type will cause the conversion of an Erlang
iolist() into an ErlIOVec (and its pointer will be passed to
the C function).
There are three rules for properly handling the refcount of binaries passed to, or returned from, the C side through a FFI call.
when a binary is received as argument (either directly, or inside
an ErlIOVec), and the C side needs to keep a reference, then
the refcount must be increased;
when a binary is created with driver_alloc_binary(), it will
have the refcount value of 1. It is considered to be still
referenced by the C side;
as a consequence of the previous point, if the C side wants to
return a newly-crated binary without keeping references, it must
call driver_binary_dec_refc() before returning.
As reported above, the high-level FFI API is based on type-tagged values. Type tags, however, may introduce yet another way to annotate/represent the types of Erlang function parameters — and it may become an annoying redundancy, especially now that type contracts are (probably) going to be introduced in Erlang.
Thus, the high-level FFI API should be considered highly experimental and subject to change, depending on how type contracts will allow to achieve the same tasks (see High-level API). This issue will need to be explored if/when contracts will be available in the standard Erlang/OTP distribution.
This EEP, and the proposed FFI patches (see below), do not introduce incompatibilities with the standard OTP release. However, three (possibly) relevant internal changes are required:
the driver_binary_dec_refc() function must be allowed to
reach the refcount of 0 without errors or warnings (even when
debugging). This is necessary in order to allow a C function to
create a binary, drop its references and return it to the Erlang VM
(see ‘Notes on Erlang binaries and reference counting’);
as a consequence of the previous point,
driver_binary_inc_refc() must be allowed to reach a minimum
refcount of 1 without errors or warnings (the current minimum value
is 2);
the iolist() -> ErlIOVec conversion code in io.c needs
to be exposed as a stand-alone function, to be used by the FFI.
An implementation of this EEP is available on muvara.org as a set of patches against OTP R11B-5.
The code is based on the GCC FFI library (libffi). libffi is multi-platform, can be packaged and used separately from the GCC source code, and is released under a very permissive license (compatible with the Erlang Public License). It has been used to implement the FFI interface of several applications and languages, including Python.
The current EEP implementation looks for libffi on the build system, and links the Erlang emulator against it (preferring the libffi shared library, when available). It may be a “good enough” approach, since libffi is usually pre-packaged and easily available on GNU/Linux, BSD and Solaris distributions. However, this approach may create troubles for developers that compile everything from scratch, could not install a precompiled libffi package, or just want to force static linking between the Erlang emulator and libffi. In order to address these issues, it is customary that a copy of libffi is distributed together with the host language, and possibly kept in sync with the upstream version. This is what Python actually does, and Erlang/OTP could possibly adopt the same approach depending on the developers’ feedback.
The following table reports the Erlang-to-C conversions, used for passing Erlang terms as C function call arguments.
====================== ===============================
C argument type Supported Erlang types
====================== ===============================
uchar integer()
schar integer()
ushort integer()
sshort integer()
uint integer()
sint integer()
ulong integer()
slong integer()
uint8 integer()
sint8 integer()
uint16 integer()
sint16 integer()
uint32 integer()
sint32 integer()
uint64 integer()
sint64 integer()
float float()
double float()
longdouble float()
pointer binary() | integer()
cstring binary() | integer()
nonnull binary() | integer()
size_t integer()
ssize_t integer()
pid_t integer()
off_t integer()
binary binary()
erliovec iolist()
====================== ===============================
The following table reports the C-to-Erlang conversions, used for converting C function return values into Erlang terms.
====================== ===============================
C return type Resulting Erlang type
====================== ===============================
uchar integer()
schar integer()
ushort integer()
sshort integer()
uint integer()
sint integer()
ulong integer()
slong integer()
uint8 integer()
sint8 integer()
uint16 integer()
sint16 integer()
uint32 integer()
sint32 integer()
uint64 integer()
sint64 integer()
float float()
double float()
longdouble float()
pointer integer()
cstring integer()
nonnull integer()
size_t integer()
ssize_t integer()
off_t integer()
pid_t integer()
binary binary()
====================== ===============================
Copyright (C) 2007 by CRS4 (Center for Advanced Studies, Research and Development in Sardinia) - http://www.crs4.it/
Author: Alceste Scalas <alceste (at) crs4 (dot) it>
This EEP is released under the terms of the Creative Commons Attribution 3.0 License. See http://creativecommons.org/licenses/by/3.0/