[erlang-questions] Updates, lenses, and why cross-module inlining would be nice

[Richard A. O'Keefe]

On 2/12/2015, at 7:19 pm, Michael Truog <mjtruog@REDACTED> wrote:

> On 12/01/2015 08:06 PM, Richard A. O'Keefe wrote:
>> On 1/12/2015, at 8:42 am, Michael Truog <mjtruog@REDACTED> wrote:
>>> The header approach is preferable to make the dependency problems
>>> a compile-time concern, rather than creating a runtime failure when
>>> attempting a module update.  It is important to not crash running source
>>> code due to errors.
>> It *would* be "preferable to make the dependency problems
>> a compile-time concern".  But using headers DOES NOT DO THAT.
> The lens.hrl created from your lens.erl file does due to containing only
> functions, that get dropped into the module that includes the header file,
> making them look like local functions within the module.

No, that does NOT "make the dependency problems a compile-time
concern".  It really doesn't.  Your lens.hrl defines functions,
and mine defines macros.  That doesn't make any difference.
Either way, this is what CREATES untracked dependencies!

T1. Module A includes header H.
    Module A is compiled.
T2. Module B includes header H.
    Module B is compiled.
(:-) Modules A and B are compatible.
T3. Header H is changed.
(:-( Modules A and B are still compatible but out of date.
T4. Module B is recompiled.
!$#@ Modules A and B are now incompatible
     AND THE SYSTEM DOES NOT KNOW THIS.

Now *if* you are diligent in using erlc's -M* options
to create/update your Makefiles, you can ensure that
both A and B will be automatically updated (when you get
around to asking for updates).

But there is no reason why erlc -M... couldn't track
-import_static just as well as -include, and ensuring that
it does would be an important part of implementing -import_static.

>   I have found this approach useful
> elsewhere.

The issue is not "is it useful".  I don't see how anyone
could argue against that.  The issue is "is it BETTER than
cross-moduling inlining" and "is it SAFER" than cross-module
inlining and the answers are NO and NO.

>> Nor does the import_static approach "crash running source code
>> due to errors."  import_static has two purposes:
>> (1) to permit cross-module inlining and type inference safely;
>> (2) to detect a clashing update *before* changing the state of
>> the system in order to *avoid* crashing running source code.
> The two (high-level) approaches I see for implementing import_static is
> where either:
> 1) modules are made permanent to make sure the modules do not change,
> so functions may be inlined between modules safely
> 2) modules are updated in groups and the groups are created by the
> inlining between modules

And in this respect there is no difference between
cross-module inlining and headers.

Except that (2) is a slightly weird way to put it.
Updates should be based on the *dependency graph* just
as updates triggered by changes to headers or to yecc
and/or leex or to parse transforms should be based on
the dependency graph.  Put thus, (2) is what I had in
mind.

> If #1 is the approach, then it should be important to have an export_static
> statement to explicitly allow module functions to be inlined, to avoid other
> developers locking your module from future changes due to their
> import_static usage.  Using the inline compile directive for this purpose
> would be overloading its meaning, since it can be necessary to inline an
> exported function for local use within a module.

If you want a function that is inlined within a module but not
inlined when called outside, have two functions.
> 
> If #2 is the approach, then the atomic update unit is probably best defined
> as an Erlang application.

There are two separate things here: *recompilation* and *reloading*.

>> It is precisely the header approach which *creates* the potential
>> for runtime problems due to version skew issues NOT BEING NOTICED.
> if you are relying on local functions changing instead of macros, the
> beam code is changing in a very noticeable way, the same as functions
> being modified.

Let's take a concrete example.

Module A and B use lens.hrl.
Initially, lens.hrl uses a two-function representation of
lenses.  Then lens.hrl is revised to use a three-function
representation of lenses.  Yes, the BEAM code for module B
has changed.  But so what?  It could change for all sorts
of reasons having NOTHING to say about interface compatibility
between modules A and B.  What module A needs to know is not
that some other module has different BEAM code but that *it*
should have different BEAM code because a header that *it*
depends on has changed.

And here we come to the heart of the problem, and it's long
past time this was fixed.  When you compile a .erl file,
the .beam file includes the name of the source file,

    BUT IT DOES NOT INCLUDE THE NAMES OF ANY HEADER FILES
    THAT THE OBJECT CODE DEPENDS ON.

If .beam files included header file names (and timestamps or
checksums), then it would be possible to ask "is module A
now out of date"?

> I understand.  My main concern is avoiding module inlining forcing
> modules to be permanent (unable to be updated during runtime)
> since that problem could easily grow in an Erlang system to negate
> the hot-code loading benefit Erlang normally provides, making its
> use harder to justify.  You can find an actor model implementation
> in many programming languages, but Erlang's fault-tolerance
> features are what sets it apart, so it is important to preserve those.

Completely agreed.

It's also relevant that cross-module inlining is of little
benefit unless the compiler inlines (fun (...) -> ... end)(...)
and currently it doesn't appear to do so.

>> If the modules in question are tested *separately*,
>> the tests don't help.  If you test the modules *together*,
>> you might as well *reload* them together.
> Then that seems to favor the approach #2 mentioned above.

Yes.

Oh, I should point out that -include files defining a bunch
of functions *do* address many of my reasons for disliking
preprocessor use.  But not the untracked dependencies problem.