Greetings all;<div><br></div><div>Sorry for the lengthy post -- TL;DR, see bottom. I asked a rather condensed version of this on the #erlang IRC channel last week; the consensus was that it should be asked here, so I put together some further notes. I previously attempted (last week) to send this message to this list to no avail, so forgive me if it somehow does wind up as a double post.</div>
<div><br></div><div><b><font class="Apple-style-span" size="4">Overview</font></b></div><div><br></div><div>I'm not sure if it's something I'm missing, but the docs seem to suggest that custom code loaders are supported by the standard ERTS (from <a href="http://www.erlang.org/doc/man/erl_prim_loader.html">erl_prim_loader(3)</a>):</div>
<div><br></div><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="arial, helvetica, sans-serif">The </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">-loader Loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> command line flag can be used to choose the method used by the </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">erl_prim_loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif">. Two </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">Loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> methods are supported by the Erlang runtime system: </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">efile</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> and </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">inet</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif">. If another loader is required, then it has to be implemented by the user. The </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">Loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> provided by the user must fulfill the protocol defined below, and it is started with the </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">erl_prim_loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> by evaluating </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">open_port({spawn,Loader},[binary]).</font></span> </blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
… </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="arial, helvetica, sans-serif">The following protocol must be followed if a user provided loader port program is used. The </font><span class="code" style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">Loader</font></span><font class="Apple-style-span" face="arial, helvetica, sans-serif"> port program is started with the command </font><font class="Apple-style-span" face="'courier new', monospace"><span class="code" style="font-weight: normal; ">open_port({spawn,Loader},[binary])</span>.</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> The protocol is as follows:</font></blockquote>
<div class="example" style="background-color: rgb(238, 238, 255); padding-top: 0px; padding-right: 10px; padding-bottom: 0px; padding-left: 10px; "><pre style="font-weight: normal; "><font class="Apple-style-span" face="'courier new', monospace">Function Send Receive
-------------------------------------------------------------
get_file [102 | FileName] [121 | BinaryFile] (on success)
[122] (failure)
stop eof terminate</font></pre></div><div><br></div><div><b><font class="Apple-style-span" size="4">Trial Run</font></b></div><div><br></div><div>So far, so good, right? Looks like there's a fairly well-defined means of implementing a code loader port. So let's try it…</div>
<div><br></div><div><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace"><b>annabel:loaders ryan$</b> <b>cat test_loader.c</b><br>#include <stdlib.h><br>#include <stdio.h><br>#include <unistd.h><br>#include <string.h></font> </blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">#define BUFSIZE 10 * 1024<br>extern int errno;</font> </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">int main(int argc, char * const argv[])<br>{<br> char req;<br> char readbuf[BUFSIZE];<br> char fname[BUFSIZE + 1];<br> ssize_t nread;<br> <br>
fprintf(stderr, "**%s:: received %d arguments\n", argv[0], argc-1);<br> for (int i=1; i<argc; ++i)<br> {<br> fprintf(stderr, "**%s:: argument %d: %s\n", argv[0], i-1, argv[i]);<br> }<br>
<br> while (1)<br> {<br> /* use unbuffered I/O [read/write(2) v. fread/fwrite(3)] */<br> nread = read(0, (void *)&req, sizeof(char));<br> if (nread == 0)<br> {<br> fprintf(stderr, "**%s:: stdin at eof, terminating...\n", argv[0]);<br>
exit(0);<br> } else if (nread < 0) {<br> perror("*** error reading port request");<br> exit(-errno);<br> } else if (req != 'f') {<br> fprintf(stderr, "**%s:: unknown mode ('%c') requested\n",<br>
argv[0], req);<br> write(1, (void *)"z", sizeof(char));<br> } else {<br> nread = read(0, (void *)&readbuf, sizeof(char) * BUFSIZE);<br> strncpy(fname, readbuf, nread + 1);<br>
fprintf(stderr, "**%s:get_file(\"%s\")\n", argv[0], fname);<br> write(1, (void *)"z", sizeof(char));<br> }<br> }<br> <br> return 0;<br>}<br><b>annabel:loaders ryan$</b> <b>cat test_loader.erl</b><br>
-module (test_loader).<br>-export ([get_file/1]).<br>-export ([start/0, start/1, interact/1]).</font> </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">-spec get_file(string() | atom()) -> binary() | failure.<br>-spec start() -> atom().<br>-spec interact(binary()) -> {char(), binary()}.</font> </blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">get_file(Filename) when is_atom(Filename) -><br> get_file(atom_to_list(Filename));<br>get_file(Filename) -><br> Req = list_to_binary([$f|Filename]),<br>
{Flag, BinaryFile} = interact(Req),<br> case Flag of<br> $z -> failure;<br> $y -> BinaryFile<br> end.</font> </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">start() -> sta rt(test_loader).<br>start(Loader) -><br> Port = open_port({spawn, Loader}, [binary]),<br> register(test_loader, Port).</font> </blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">interact(Req) -><br> case whereis(test_loader) of<br> undefined -><br> start(),<br> interact(Req);<br> Port -><br>
Port ! {self(), {command, Req}},<br> receive<br> {Port, {data, <<Flag, Binary/binary>>}} -> {Flag, Binary}<br> end<br> end.<br><b>annabel:loaders ryan$</b> <b>CFLAGS='-std=c99 -Wall -pedantic' make test_loader</b><br>
cc -std=c99 -Wall -pedantic test_loader.c -o test_loader<br><b>annabel:loaders ryan$</b> <b>PATH=.:$PATH erl</b><br>Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]<br>
Eshell V5.8.4 (abort with ^G)<br>1> c(test_loader).<br>{ok,test_loader}<br>2> test_loader:start().<br>true<br>3> **test_loader:: received 0 arguments</font> </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
</blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">3> test_loader:get_file("testing").<br>**test_loader:get_file("testing")<br> failure<br>4> <br>
User switch command<br> --> q<br>**test_loader:: stdin at eof, terminating...</font> </blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace"><b>annabel:loaders ryan$</b> <b>PATH=.:$PATH erl -loader test_loader</b></font></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">{"cannot start loader",{function_clause,[{erl_prim_loader,start_it,["test_loader",none,<0.2.0>,[]]}]}}</font></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">{"init terminating in do_boot",{function_clause,[{erl_prim_loader,start_it,["test_loader",none,<0.2.0>,[]]}]}}</font></blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">(no error logger present) error: "Error in process <0.3.0> with exit value: {function_clause,[{erl_prim_loader,start_it,[\"test_loader\",none,<0.2.0>,[]]}]}\n"</font></blockquote>
<blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace"><br></font></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">Crash dump was written to: erl_crash.dump</font></blockquote><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">init terminating in do_boot ()</font></blockquote></div><div><br></div><div><b><font class="Apple-style-span" size="4">Error Location</font></b></div>
<div><br></div><div>Referring to <font class="Apple-style-span" face="'courier new', monospace">erl_prim_loader</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> (</font><font class="Apple-style-span" face="'courier new', monospace">$OTP_SRC/erts/preloaded/erl_prim_loader.erl</font><font class="Apple-style-span" face="arial, helvetica, sans-serif">), the error reported soon reveals itself in </font><font class="Apple-style-span" face="'courier new', monospace">start_it/4 </font><font class="Apple-style-span" face="arial, helvetica, sans-serif">(presumably where the </font><font class="Apple-style-span" face="'courier new', monospace">open_port/2</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> call referenced in </font><font class="Apple-style-span" face="'courier new', monospace">erl_prim_loader(3)</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> would be made):</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
<font class="Apple-style-span" face="'courier new', monospace">start_it("ose_inet"=Cmd, Id, Pid, Hosts) -><br> …<br> start_it("inet", Id, Pid, Hosts);<br>start_it("inet", Id, Pid, Hosts) -><br>
…<br> loop(State, Pid, []);<br>start_it("efile", Id, Pid, _Hosts) -><br> …<br> loop(State, Pid, []).</font></blockquote><div><br></div><div><b>Issue 1: </b>No clauses are defined here for custom code loaders, only <font class="Apple-style-span" face="'courier new', monospace">"ose_inet"</font>, <font class="Apple-style-span" face="'courier new', monospace">"inet"</font> and <font class="Apple-style-span" face="'courier new', monospace">"efile"</font>.</div>
<div><br></div><div><b><font class="Apple-style-span" size="4">Further Issues</font></b></div><div> </div></div><div>Continuing down the file, further issues crop up. <font class="Apple-style-span" face="'courier new', monospace">erl_prim_loader:loop/3 </font><font class="Apple-style-span" face="arial, helvetica, sans-serif">relies on a corpus of helper functions (all defined in the same module) to do its dirty work; several of these have issues.</font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><b>Issue 2:</b> Several functions only accept </font><font class="Apple-style-span" face="'courier new', monospace">efile</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> or </font><font class="Apple-style-span" face="'courier new', monospace">inet</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> for </font><font class="Apple-style-span" face="'courier new', monospace">#state.loader</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> -- any other value will cause a </font><font class="Apple-style-span" face="'courier new', monospace">function_clause</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> error:</font></div>
<div><ul><li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_get_file/3</span></li><li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_list_dir/2</span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_get_cwd/2</span></li><li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_stop/2</span></li>
<li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_exit/3</span></li><li><span class="Apple-style-span" style="font-family: 'courier new', monospace; ">handle_timeout/2</span></li>
</ul><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><b>Issue 3:</b> Two other functions only accept </font><font class="Apple-style-span" face="'courier new', monospace">efile</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> for </font><font class="Apple-style-span" face="'courier new', monospace">#state.loader</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> -- same error as above:</font></div>
<ul><li><font class="Apple-style-span" face="'courier new', monospace">handle_set_primary_archive/4</font></li><li><font class="Apple-style-span" face="'courier new', monospace">handle_release_archives/1</font></li>
</ul><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><b>Issue 4:</b> </font><font class="Apple-style-span" face="'courier new', monospace">handle_get_files/4</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> will fail for any loader besides </font><font class="Apple-style-span" face="'courier new', monospace">efile</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> that sets </font><font class="Apple-style-span" face="'courier new', monospace">#state.multi_get</font><font class="Apple-style-span" face="arial, helvetica, sans-serif"> to t</font><font class="Apple-style-span" face="'courier new', monospace">rue</font><font class="Apple-style-span" face="arial, helvetica, sans-serif">.</font></div>
</div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><br></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif" size="4"><b>TL;DR Summary</b></font></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><b><br></b></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">I can't see how this shipped for so many versions as "broken" as it seems -- I can't have been the first one to attempt to implement a custom code loader. Have I just entirely missed the boat on how this is done, or are these actually bugs? </font><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">Most importantly, is there any interest in a patch that fixes these apparent issues?</span></div>
<div><font class="Apple-style-span" face="arial, helvetica, sans-serif"><b><br></b></font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Regards,</font></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">Ryan</font></div>