[erlang-questions] Updates, lenses, and why cross-module inlining would be nice

[Michael Truog]

On 11/29/2015 05:58 PM, Richard A. O'Keefe wrote:
> On 27/11/2015, at 6:30 pm, Michael Truog <mjtruog@REDACTED> wrote:
>> Having it be opt-in does sound good, but it should complicate
>> hot-code loading, making a module update possibly fail due to the
>> import_static dependency on an external module.
> We already have precisely this problem with headers.
> Header dependencies are not tracked by the code loading
> support.  If modules A and B rely on header C to
> describe their interface, and C changes and B is updated
> to match, the fact that A doesn't match any more is not
> detected.
>
> At least with an import_static dependency *stated* and
> right there in the .beam file (as it should be) you can
> *tell* when you have a problem.
The header approach is preferable to make the dependency problems
a compile-time concern, rather than creating a runtime failure when
attempting a module update.  It is important to not crash running source
code due to errors.
>
>>   If module 'a' uses
>> import_static on module 'b', does the inlining (actual source code) of
>> module 'a' change during runtime if module 'b' changes?  When the
>> module 'b' update indirectly alters module 'a', does its module
>> version not change?  If you were allowed to do an update of a
>> module used for inlining, it could create a ripple-effect in the system,
>> making the process more brittle and prone to failure, due to the
>> lack of isolation.
> Headers are brittle and prone to failure in precisely this
> way right now, it's just that the system doesn't notice.
>
> Hmmm.  The lens module could perfectly well be a header.
> Instead of
>
> get({G,_,_}, X) -> G(X).
>
> in a module, we'd have
>
> -define(GET(Lens, Datum), case Lens of {G,_,_} -> G(Datum) end).
>
> and so on.  Would this change the actual number or character of
> dependencies?  No.  What would be the effect of changing a
> lens.hrl file instead of a lens.erl file?  Just the same breakage
> except that the system wouldn't notice and would not be able to
> reject a load telling you something would have broken.
I was thinking about a slightly different approach, which I adapted your
module to at https://github.com/okeuday/lens/ .  The header file is at
https://github.com/okeuday/lens/blob/master/include/lens.hrl .
The main changes can be summarized as:
1) change all functions to have a "lens_" prefix
2) add a compile nowarn_unused_function directive with all the interface
    functions (comment: unfortunately the F/A syntax is not supported here
    right now, but it is supported in the inline compile directive)
3) move the example to an eunit test at
     https://github.com/okeuday/lens/blob/master/test/lens_test.erl

I put the module example and the header example in the same eunit test
to show that the main difference visually is using a "_" instead of a ":".
The inlining creates warnings due to terms not being used from the header
file, which is annoying, but shows that the inlining works (I don't believe
there is a way to turn off that warning, currently).  I was using rebar2 to
build/test.

The point of this, is to show the same source code can be used, with
inlining, and that the potential for breakage can be handled with testing
all the functions.  A lens interface should not change a whole lot, so it
can be a dependable interface that is trusted.

>
> It would of course be possible to provide a parse transform
> for any specific module to accomplish the same effect as
> cross-module inlining, but this has *precisely* the same
> dependency effect as cross-module inlining or headers.
>
> "We have already determined what you are madam, now we're
> haggling over the price."
>
> I believe that dependence on a parse transform or a header
> should be tracked in .beam files and that if you try to
> load a .beam file the system ought to be able to tell you
> "this has a dependency which has since been changed, why
> not recompile first?"
This would be interesting, but would likely need to use a hash of the
parse transform contents and header file contents.  I don't think it
would be very helpful though, since parse transforms and header files
are only used during compilation and the build should be checking
timestamps to see if files change.  If it tracked the preprocessor
input, it could say "yes, this preprocessor input changed" or
"no, this preprocessor input did not change", but you can't
determine what the changes were from the .beam addition or
anything beyond that.  What do you do with the information?

>> If instead the modules that are inlined are marked "permanent" and
>> prevented from being updated in the future, then there is a reason
>> to not use inlining
> I am not with you.  How does making inlining *safe* become a
> reason to *not* use it?
>
> It's not as if I'm proposing anything that wasn't already known
> in the Lisp world back in the 70s.
If modules are marked permanent due to module inlining, it will be preventing
modules from changing in the future.  If it is done as you have described, it would
be controlled by higher-level source code, preventing lower-level source code
from changing, making development less flexible.  It works fine, until it doesn't,
then you have people renaming modules to avoid the problem, and trying to
maintain more than 1 copy of the source code.  I see that as a way to cause
problems, not solve them.
>> You may be able to make a lens implementation that relies on
>> header file usage,
> That's rather like saying "You could avoid the problems of
> playing Russian Roulette with a pistol by using a fully
> loaded shotgun instead."
>
>> which would allow all the functions to be within a
>> single module and allowed to be inlined there.  That approach
>> wouldn't cause problems for module updating in the future.
> It would cause *precisely* the same problems that headers
> have always (at least potentially) caused.
>
> m% cat goo.hrl
> -define(GOO, 42).
>
> m% cat foo.erl
> -module(foo).
> -export([foo/0]).
> -include("goo.hrl").
>
> foo() ->
>      boo:boo(?GOO).
>
> m% erlc foo.erl
> m% ed goo.hrl
> s/42/137/
> w
> q
> m% cat goo.hrl
> -define(GOO, 137).
>
> m% cat boo.erl
> -module(boo).
> -export([boo/1]).
> -include("goo.hrl").
>
> boo(?GOO) ->
>      ok.
> m% erlc erlc boo.erl
> m% erl
>
> 1> l(boo).
> {module,boo}
> 2> l(foo).
> {module,foo}
> 3> foo:foo().
> ** exception error: no function clause matching boo:boo(42) (boo.erl, line 5)
> 4> c(boo).
> {ok,boo}
> 5> c(foo).
> {ok,foo}
> 6> foo:foo().
> ok
>
> RIGHT NOW you can get mysterious errors that go away
> when you recompile things, due to the use of headers.
> import_static doesn't increase the problems, or the
> amount of work you have to do to fix them, or the nature
> of that work.  All it does is give the system a chance
> to *notice* that the problem already exists.
Yes.  I believe this doesn't become a concern when tests are provided.

>
> ANY use of information that is not physically present in
> the source file, whether it comes from using Leex or Yecc
> or the preprocessor or a parse transform or anything else
> is a *dependency* that needs to be tracked.
Ok, but what runtime problem does this solve?