[erlang-bugs] bug in code:lib_dir

[Joe Armstrong]

On Tue, Sep 18, 2012 at 1:57 PM, Serge Aleynikov <serge@REDACTED> wrote:
> Joe,
>
> When I wrote the code supporting ERL_LIBS that got included in the code
> server (*), it was explicitly designed to iteratively check for paths
> only containing "ebin" child directories (see the
> code_server:get_user_lib_dirs_1/0) before adding them to the loader
> path.  Therefore I wouldn't consider the behavior you mentioned to be a
> bug but rather a feature.
>
> Initially I wrote it in a way that checked for directories containing
> "*.app" files, and included those.  However, I found cases when certain
> libraries didn't come with "*.app" files, but merely contained
> "ebin/*.beam" bytecode files, so the ebin directory checking was more
> reliable and conventional.
>
> In your case, since 'b' application doesn't seem to have neither b.app
> nor b/ebin.  Should it be considered a valid code path (there's nothing
> to load for the code server)?  If so, what is the right selection
> criteria that would avoid bloating the loader's path?

Well the problem is that code:lib_dir(a) is called when
the compiler tries to compile a file containing an include_lib
directive like

    -include_lib("a/include/foo.hrl").

If there really is a file called "a/include/foo.hrl" then it seems
rather harsh to fail the compilation.

rebar in particular only creates an ebin direction as a result of a
compile - I googled a bit and found many cases where a
missing include file message was worrying people - and I think this
was the reason. In the project I was looking at
this bug occurred and was fixed with a work-around.

I'm not sure about "code loader path bloat" if you trust
the LIB_DIRS env variable is correctly set. I'm not sure if
you recursively scan all directories below  the LIB_DIRS
but I guess just one level would be ok.

The problem also happens (I think) if git doesn't remake an
empty ebin directory.

The problem is in find_lib_dir (line 637) of epp.erl

Actually the compile error message could be more helpful

    can't find include lib "aaa/include/d.yrl"

It's true - but confused me for a hour - why could't it find
this file - ERL_LIBS had aaa in it's path - if the message had been

      can't find include lib "aaa/include/d.yrl"
      (possible reason aaa/ebin is missing)

I would have guessed what was going wrong.

The root of the problem is that you can't identify app directories -
sooner or later this must be fixed - I think
there should be a proper manifest.json (or something) in
top level of the app - this and only this would identify this as an app.

Until then we need various work-arounds. Probably easiest
to change epp so is calls something other than code:lib_dir
when resolving the filename

Cheers

/Joe





>
> Serge
>
> (*) http://www-rohan.sdsu.edu/doc/R12B/lib/kernel-2.12.3/doc/html/notes.html
>
> On 9/18/2012 4:14 AM, Joe Armstrong wrote:
>>   > code:lib_dir(b)
>>    {error,bad_name}
>>
>> Fails if the b sub-directory does not contain an ebin directory.
>> This means things like include_lib("b/include/foo.hrl") will fail
>> even though the file b/include/foo.hrl exists.
>>
>>
>> Suppose I have a directory structure like this:
>>
>>      /home/joe/deps/a/src
>>                      /include
>>                      /ebin
>>                    /b/src
>>                      /include/foo.hrl
>>
>>
>> a and b are Erlang applications - note that b has no ebin directory
>>
>> Then I say
>>
>>     > export ERL_LIBS=/home/joe/deps
>>     > erl
>>
>> Then
>>
>>    > code:lib_dir(a).
>>    "/home/joe/deps/a/"
>>
>> But
>>
>>    > code:lib_dir(b)
>>    {error,bad_name}
>>
>> Now suppose a.erl contains the line
>>
>>   -include_lib("b/include/foo.hrl")
>>
>>   I'll now get an error. The code server calls lib_dib(b) to resolve
>> the file name
>> reference. This fails and the system concludes (incorrectly) that
>> b/include/foo.hrl does not
>> exist.
>>
>>   If I add an empty ebin directory to b then the error goes away.
>>
>>   If you create src, ebin, include directories by hand you will never
>> see this error.
>>
>>   But if you use rebar then the ebin directory is not created when you do
>>   > rebar create-app appid=<name>
>>
>>   The ebin directory is created when you first run rebar compile.
>>
>>   So recursive includes (a includes b/include and b includes
>> a/include) can possibly
>>   fail if no ebin directory is created.
>>
>> Cheers
>>
>> /Joe
>> _______________________________________________
>> erlang-bugs mailing list
>> erlang-bugs@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-bugs
>>