[erlang-bugs] bug in code:lib_dir

[Serge Aleynikov]

+1.

On 9/18/2012 3:39 PM, Fred Hebert wrote:
> It sounds like you could solve the problem quite easily by adding an app
> file to your directory, transforming it into a library application,
> assuming the following project structure:
> 
> lib_dir/ebin/my_app.app
> lib_dir/include/my_app.hrl
> 
> where my_app.app looks as follows:
> 
> {application, my_app,
>  [{description, "useful header files for Joe"},
>   {vsn, "1.0.0"},
>   {modules, []}]}.
> 
> This should give you an entirely empty library application -- with no
> beam code, but a .app file is present. This makes it visible to
> ERL_LIBS, usable as a dependency with rebar, visible and manageable in
> releases, etc.
> 
> 
> On 12-09-18 2:54 PM, Joe Armstrong wrote:
>> On Tue, Sep 18, 2012 at 1:57 PM, Serge Aleynikov <serge@REDACTED>
>> wrote:
>>> Joe,
>>>
>>> When I wrote the code supporting ERL_LIBS that got included in the code
>>> server (*), it was explicitly designed to iteratively check for paths
>>> only containing "ebin" child directories (see the
>>> code_server:get_user_lib_dirs_1/0) before adding them to the loader
>>> path.  Therefore I wouldn't consider the behavior you mentioned to be a
>>> bug but rather a feature.
>>>
>>> Initially I wrote it in a way that checked for directories containing
>>> "*.app" files, and included those.  However, I found cases when certain
>>> libraries didn't come with "*.app" files, but merely contained
>>> "ebin/*.beam" bytecode files, so the ebin directory checking was more
>>> reliable and conventional.
>>>
>>> In your case, since 'b' application doesn't seem to have neither b.app
>>> nor b/ebin.  Should it be considered a valid code path (there's nothing
>>> to load for the code server)?  If so, what is the right selection
>>> criteria that would avoid bloating the loader's path?
>> Well the problem is that code:lib_dir(a) is called when
>> the compiler tries to compile a file containing an include_lib
>> directive like
>>
>>      -include_lib("a/include/foo.hrl").
>>
>> If there really is a file called "a/include/foo.hrl" then it seems
>> rather harsh to fail the compilation.
>>
>> rebar in particular only creates an ebin direction as a result of a
>> compile - I googled a bit and found many cases where a
>> missing include file message was worrying people - and I think this
>> was the reason. In the project I was looking at
>> this bug occurred and was fixed with a work-around.
>>
>> I'm not sure about "code loader path bloat" if you trust
>> the LIB_DIRS env variable is correctly set. I'm not sure if
>> you recursively scan all directories below  the LIB_DIRS
>> but I guess just one level would be ok.
>>
>> The problem also happens (I think) if git doesn't remake an
>> empty ebin directory.
>>
>> The problem is in find_lib_dir (line 637) of epp.erl
>>
>> Actually the compile error message could be more helpful
>>
>>      can't find include lib "aaa/include/d.yrl"
>>
>> It's true - but confused me for a hour - why could't it find
>> this file - ERL_LIBS had aaa in it's path - if the message had been
>>
>>        can't find include lib "aaa/include/d.yrl"
>>        (possible reason aaa/ebin is missing)
>>
>> I would have guessed what was going wrong.
>>
>> The root of the problem is that you can't identify app directories -
>> sooner or later this must be fixed - I think
>> there should be a proper manifest.json (or something) in
>> top level of the app - this and only this would identify this as an app.
>>
>> Until then we need various work-arounds. Probably easiest
>> to change epp so is calls something other than code:lib_dir
>> when resolving the filename
>>
>> Cheers
>>
>> /Joe
>>
>>
>>
>>
>>
>>> Serge
>>>
>>> (*)
>>> http://www-rohan.sdsu.edu/doc/R12B/lib/kernel-2.12.3/doc/html/notes.html
>>>
>>> On 9/18/2012 4:14 AM, Joe Armstrong wrote:
>>>>    > code:lib_dir(b)
>>>>     {error,bad_name}
>>>>
>>>> Fails if the b sub-directory does not contain an ebin directory.
>>>> This means things like include_lib("b/include/foo.hrl") will fail
>>>> even though the file b/include/foo.hrl exists.
>>>>
>>>>
>>>> Suppose I have a directory structure like this:
>>>>
>>>>       /home/joe/deps/a/src
>>>>                       /include
>>>>                       /ebin
>>>>                     /b/src
>>>>                       /include/foo.hrl
>>>>
>>>>
>>>> a and b are Erlang applications - note that b has no ebin directory
>>>>
>>>> Then I say
>>>>
>>>>      > export ERL_LIBS=/home/joe/deps
>>>>      > erl
>>>>
>>>> Then
>>>>
>>>>     > code:lib_dir(a).
>>>>     "/home/joe/deps/a/"
>>>>
>>>> But
>>>>
>>>>     > code:lib_dir(b)
>>>>     {error,bad_name}
>>>>
>>>> Now suppose a.erl contains the line
>>>>
>>>>    -include_lib("b/include/foo.hrl")
>>>>
>>>>    I'll now get an error. The code server calls lib_dib(b) to resolve
>>>> the file name
>>>> reference. This fails and the system concludes (incorrectly) that
>>>> b/include/foo.hrl does not
>>>> exist.
>>>>
>>>>    If I add an empty ebin directory to b then the error goes away.
>>>>
>>>>    If you create src, ebin, include directories by hand you will never
>>>> see this error.
>>>>
>>>>    But if you use rebar then the ebin directory is not created when
>>>> you do
>>>>    > rebar create-app appid=<name>
>>>>
>>>>    The ebin directory is created when you first run rebar compile.
>>>>
>>>>    So recursive includes (a includes b/include and b includes
>>>> a/include) can possibly
>>>>    fail if no ebin directory is created.
>>>>
>>>> Cheers
>>>>
>>>> /Joe
>>>> _______________________________________________
>>>> erlang-bugs mailing list
>>>> erlang-bugs@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-bugs
>>>>
>> _______________________________________________
>> erlang-bugs mailing list
>> erlang-bugs@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-bugs
>