Hot loading and preprocessor [was: Re: the Hopwood design process]

[Richard A. O'Keefe]

I wrote:
	> There really isn't anything that can be done with the processor
	> [should have been "preprocessor"] that
	> could not be done better without it.	

Kostis Sagonas <kostis@REDACTED> replied:
	Quite a strong statement, but probably true.

I have tried the exercise of rewriting a couple of thousand lines
of OTP sources, about 4 years ago.  It's true all right.

I also said that
	 > he preprocessor subverts
	 > the module system and causes dependencies between source units
	 > that are not and cannot be tracked by the run time system.
	
	I am not sure I get this part though. AFAIK, the Erlang run time
	system does not track any dependencies between modules anyway --
	modules can be hot (re)loaded on an individual basis and you get
	"reload-and-pray" semantics when you do so and forget to update
	modules that these modules depend on.
	
I didn't say that the existing system does track module dependencies.
It could, and I think it definitely should.  As a regular reader of
comp.risks, I prefer "tentatively reload and check" to "reload and pray"
*if* the checks can be reliable.  But that relies on *having* modules,
and having language constructs that respect modules.

Let me briefly recapitulate another old proposal:

    -use_early(Module).

	means that this module depends on Module; calls with Module: as
	a module prefix should be accepted without comment; the
	dependency between this module and Module is so close that the
	compiler may inline definitions from Module; this module may
	be reloaded without necessarily reloading Module, but if
	Module is reloaded a corresponding version of this module
	should be loaded at the same time.

    -use_late(Module).

	means that this module depends on Module; calls with Module: as
	a module prefix should be accepted without comment; the compiler
	should assume that Module may be reloaded without this module
	being reloaded.  (That doesn't preclude inlining, but it would
	seem to require on-the-fly recompilation of some kind.)

    -use_early(Module, [F1/N1, ..., Fk/Nk]).

	means the same as -use_early(Module) plus in addition any call
	to a function (or pattern) in Module other than one listed in
	a declaration like this should provoke a compiler warning.
	It is an error if Module does not define and export F1/N1 ... Fk/Nk,
	to be reported no later than the time that this module and Module
	are both loaded.

    -use_late(Module, [F1/N1, ..., Fk/Nk]).

	means the same as -use_late(Module) plus in addition any call
	to a function (or pattern) in Module other than one listed in
	a declaration like this should provoke a compiler warning.
	If Module does not define and export some Fi/Ni, that need not
	be reported until Module:Fi/Ni is called.

    -import_early(Module, [F1/N1, ..., Fk/Nk]).

	means the same as -use_early(Module, [F1/N1, ..., Fk/Nk])
	plus in addition any function listed in a declaration like
	this may be used without a module prefix; it is an error if
	any Fi/Ni is declared in this module.

    -import_late(Module, [F1/N1, ..., Fk/Nk]).

	plus in addition any function listed in a declaration like
	this may be used without a module prefix; it is an error if
	any Fi/Ni is declared in this module.

The modules of an application are often (though not always) reloaded
as a unit, so it makes sense for modules in an application to be
able to -use_early or -import_early each other.  In fact, it even
makes sense for the compiled modules of an application to be held in
a single file, so that it is _easier_ to reload the whole thing than
to reload just one file.

(End of recapitulation.)

	However, even if it did, dependencies could take preprocessing into
	account.  For example, the compiler+loader could easily record
	information of the form "the code of all this set of modules has
	a dependency on these record declarations".
	Even today, this info is present when compiling with +debug_info.
	
Well, not quite.  You see the preprocessor can create *negative*
dependencies.  With the declarations above, we can create a
dependency that says

    module M1 relies on module M2 providing F/N.

With the preprocessor, we can create dependencies that say

    file F1 relies on file F2 *not* defining ?X.

More precisely,

    *use* such and such of file F1 relies on file F2 not defining ?X

and it is possible for one use of F1 to rely on F2 HAVING ?X and
another use of F1 in the same application to rely on F2 NOT having ?X.

Also, the things that the dependencies relate are (uses of) files,
not modules.  Without the preprocessor, it's just modules.

One reason this matters is that the module name space is under Erlang's
control, but the file name space is not.

	I certainly do not want to defend the preprocessor, but I do not
	really see how the preprocessor is to blame for what is being
	discussed here...  

I'm not sure what you mean by "what is being discussed here".
I hope I've explained in a bit more detail what the preprocessor
does to dependencies.

	How are records different than e.g. changing
	the types of data structures that some module obtains from some
	other module and manipulates?
	
This is like asking "how is a brick different from breaking an egg?"
One is a thing, the other is an activity.

The main issue is that Erlang is a language based on modules,
and records are not based on and do not respect modules.

If you obtain a data structure from some other module and manipulate it,
*as long as you use the functions exported from that module for the
purpose of such manipulation*, you should be OK.  (Until the module is
revised and reloaded, and even then abstract patterns and psi-terms can
help a lot.)

If you get a record from another module and manipulate it using
record syntax, you *have* to bypass the module system to do so.
Both this module and the other module must get the record definition
from something that IS NOT A MODULE.  (Or both modules could have their
own declaration for the record type, which also counts as bypassing the
module system.)

Don't get me wrong.
I *agree* that something *like* records are so very useful that Erlang
had to have something *like* them.
I even agree that the present approach to records is better than
nothing at all, and was worth a try if nothing better could have been
invented.

The issue is whether we could have something *like* records that meshes
well with the module system, or whether it's necessary to imitate C
quite so slavishly.