[erlang-questions] Code loaders other than efile and inet

Ryan Volpe ryan.volpe@REDACTED
Mon Jul 11 08:31:38 CEST 2011


Greetings all;

Sorry for the lengthy post -- TL;DR, see bottom. I asked a rather condensed
version of this on the #erlang IRC channel last week; the consensus was that
it should be asked here, so I put together some further notes. I previously
attempted (last week) to send this message to this list to no avail, so
forgive me if it somehow does wind up as a double post.

*Overview*

I'm not sure if it's something I'm missing, but the docs seem to suggest
that custom code loaders are supported by the standard ERTS (from
erl_prim_loader(3) <http://www.erlang.org/doc/man/erl_prim_loader.html>):

The -loader Loader command line flag can be used to choose the method used
> by the erl_prim_loader. Two Loader methods are supported by the Erlang
> runtime system: efile and inet. If another loader is required, then it has
> to be implemented by the user. The Loader provided by the user must
> fulfill the protocol defined below, and it is started with the
> erl_prim_loader by evaluating open_port({spawn,Loader},[binary]).

…



The following protocol must be followed if a user provided loader port
> program is used. The Loader port program is started with the command
> open_port({spawn,Loader},[binary]). The protocol is as follows:

Function          Send               Receive
-------------------------------------------------------------
get_file          [102 | FileName]   [121 | BinaryFile] (on success)
                                     [122]              (failure)
stop              eof                terminate


*Trial Run*

So far, so good, right? Looks like there's a fairly well-defined means of
implementing a code loader port. So let's try it…

*annabel:loaders ryan$* *cat test_loader.c*
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> #include <string.h>



#define BUFSIZE 10 * 1024
> extern int errno;



int main(int argc, char * const argv[])
> {
>     char req;
>     char readbuf[BUFSIZE];
>     char fname[BUFSIZE + 1];
>     ssize_t nread;
>
>     fprintf(stderr, "**%s:: received %d arguments\n", argv[0], argc-1);
>     for (int i=1; i<argc; ++i)
>     {
>         fprintf(stderr, "**%s:: argument %d: %s\n", argv[0], i-1, argv[i]);
>     }
>
>     while (1)
>     {
>         /* use unbuffered I/O [read/write(2) v. fread/fwrite(3)] */
>         nread = read(0, (void *)&req, sizeof(char));
>         if (nread == 0)
>         {
>             fprintf(stderr, "**%s:: stdin at eof, terminating...\n",
> argv[0]);
>             exit(0);
>         } else if (nread < 0) {
>             perror("*** error reading port request");
>             exit(-errno);
>         } else if (req  != 'f') {
>             fprintf(stderr, "**%s:: unknown mode ('%c') requested\n",
>                     argv[0], req);
>             write(1, (void *)"z", sizeof(char));
>         } else {
>             nread = read(0, (void *)&readbuf, sizeof(char) * BUFSIZE);
>             strncpy(fname, readbuf, nread + 1);
>             fprintf(stderr, "**%s:get_file(\"%s\")\n", argv[0], fname);
>             write(1, (void *)"z", sizeof(char));
>         }
>     }
>
>     return 0;
> }
> *annabel:loaders ryan$* *cat test_loader.erl*
> -module (test_loader).
> -export ([get_file/1]).
> -export ([start/0, start/1, interact/1]).



-spec get_file(string() | atom()) -> binary() | failure.
> -spec start() -> atom().
> -spec interact(binary()) -> {char(), binary()}.



get_file(Filename) when is_atom(Filename) ->
>     get_file(atom_to_list(Filename));
> get_file(Filename) ->
>     Req = list_to_binary([$f|Filename]),
>     {Flag, BinaryFile} = interact(Req),
>     case Flag of
>         $z -> failure;
>         $y -> BinaryFile
>     end.



start() -> sta  rt(test_loader).
> start(Loader) ->
>     Port = open_port({spawn, Loader}, [binary]),
>     register(test_loader, Port).



interact(Req) ->
>     case whereis(test_loader) of
>         undefined ->
>             start(),
>             interact(Req);
>         Port ->
>             Port  ! {self(), {command, Req}},
>             receive
>                 {Port, {data, <<Flag, Binary/binary>>}} -> {Flag, Binary}
>             end
>     end.
> *annabel:loaders ryan$* *CFLAGS='-std=c99 -Wall -pedantic' make
> test_loader*
> cc -std=c99 -Wall -pedantic    test_loader.c   -o test_loader
> *annabel:loaders ryan$* *PATH=.:$PATH erl*
> Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2]
> [async-threads:0] [hipe] [kernel-poll:false]
> Eshell V5.8.4  (abort with ^G)
> 1> c(test_loader).
> {ok,test_loader}
> 2> test_loader:start().
> true
> 3> **test_loader:: received 0 arguments



3> test_loader:get_file("testing").
> **test_loader:get_file("testing")
>                                  failure
> 4>
> User switch command
>  --> q
> **test_loader:: stdin at eof, terminating...

*annabel:loaders ryan$* *PATH=.:$PATH erl -loader test_loader*

{"cannot start
> loader",{function_clause,[{erl_prim_loader,start_it,["test_loader",none,<0.2.0>,[]]}]}}

{"init terminating in
> do_boot",{function_clause,[{erl_prim_loader,start_it,["test_loader",none,<0.2.0>,[]]}]}}

(no error logger present) error: "Error in process <0.3.0> with exit value:
> {function_clause,[{erl_prim_loader,start_it,[\"test_loader\",none,<0.2.0>,[]]}]}\n"


> Crash dump was written to: erl_crash.dump

init terminating in do_boot ()


*Error Location*

Referring to erl_prim_loader ($OTP_SRC/erts/preloaded/erl_prim_loader.erl),
the error reported soon reveals itself in start_it/4 (presumably where the
open_port/2 call referenced in erl_prim_loader(3) would be made):

start_it("ose_inet"=Cmd, Id, Pid, Hosts) ->
>>     start_it("inet", Id, Pid, Hosts);
> start_it("inet", Id, Pid, Hosts) ->
>>     loop(State, Pid, []);
> start_it("efile", Id, Pid, _Hosts) ->
>>     loop(State, Pid, []).


*Issue 1: *No clauses are defined here for custom code loaders, only
"ose_inet", "inet" and "efile".

*Further Issues*

Continuing down the file, further issues crop up. erl_prim_loader:loop/3 relies
on a corpus of helper functions (all defined in the same module) to do its
dirty work; several of these have issues.

*Issue 2:* Several functions only accept efile or inet for #state.loader --
any other value will cause a function_clause error:

   - handle_get_file/3
   - handle_list_dir/2
   - handle_get_cwd/2
   - handle_stop/2
   - handle_exit/3
   - handle_timeout/2

*Issue 3:* Two other functions only accept efile for #state.loader -- same
error as above:

   - handle_set_primary_archive/4
   - handle_release_archives/1

*Issue 4:* handle_get_files/4 will fail for any loader besides efile that
sets #state.multi_get to true.


*TL;DR Summary*
*
*
I can't see how this shipped for so many versions as "broken" as it seems --
I can't have been the first one to attempt to implement a custom code
loader. Have I just entirely missed the boat on how this is done, or are
these actually bugs? Most importantly, is there any interest in a patch that
fixes these apparent issues?
*
*
Regards,
Ryan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20110711/4bc0ee48/attachment.htm>


More information about the erlang-questions mailing list