Metaprogramming

[Yariv Sadan]

hi Jay,

Thanks for sharing your interesting thoughts about metaprogramming
with Erlang. When I first created Smerl, I knew I was just scratching
the surface of what one could do with the ability for manipulating
Erlang modules on the fly, and I was waiting to see what ideas other
people will come up with.

I have actually been using Smerl quite a lot, and over time I have
discovered I needed some higher level features from Smerl that go
beyond adding and removing functions to modules: metacurrying,
parameter embedding and module extensions. To implement these
features, Smerl performs deep surgery inside a function's abstract
form. Is that "real" metaprogramming?

Side note: when I created Smerl, I didn't mean to suggest necessarily
that Smerl *does* metaprogramming, but rather that Smerl facilitates
metaprogramming. At least with the basic Smerl features, the
programmer does the actual metaprogramming.

The techniques that Smerl use (or at least some of them) have been
done before, at least in Ulf Wiger's rdbms_codegen.erl module. Smerl
just makes such metaprogramming wizardry easier to accomplish.

What is metaprogramming? According to wikipedia:

"Metaprogramming is the writing of programs that write or manipulate
other programs (or themselves) as their data or that do part of the
work that is otherwise done at compile time during runtime. In many
cases, this allows programmers to get more done in the same amount of
time as they would take to write all the code manually.

The language in which the metaprogram is written is called the
metalanguage. The language of the programs that are manipulated is
called the object-language. The capacity of a programming language to
be its own meta-language is called reflexivity."


By this definition, Erlang is a reflexive language, because it allows
the programmer to write programs that write or manipulate other
programs. Smerl doesn't add this capability to Erlang -- it only
exposes it and makes it easier to use.

> To me metaprogramming is writing code that dynamically changes the rules
> of the language.

Using Smerl in my project, I am certainly using Erlang very
differently. I'm creating a lot of metafunctions (skeletons for
functions that are to be metacurried in runtime) and abstract modules
to be used as parents for modules that are to be generated in runtime.
Does that mean Smerl changes the rules of the language? I guess it all
depends on your definition.

I think it does :)

Best,
Yariv


On 8/23/06, Jay Nelson <jay@REDACTED> wrote:
> When I first saw the smerl post my thinking went like this:
>
>    - Normal programs execute and return a value
>    - When I program, I generate source code and run the compiler
>    - smerl is just doing normal programming
>
> This is a step up from executing, but it is programming.
>
> Then I looked on wikipedia and they call Generative Programming a
> metaprogramming technique, so a for loop that spits out source code is
> metaprogramming.  I guess it is a question of what your background is in
> making a determination like that.
>
> I progressed through programming languages in the following order:
>
> 1) Basic
> 2) APL
> 3) RPG
> 4) Prolog
> 5) Lisp / Flavors (Symbolics) / Common Lisp Object System (CLOS)
> 6) C then C++
> 7) Perl
> 8) erlang
> 9) C#
>
> I guess I learned some SNOBOL between 5 & 6 and REFAL around the same
> time.  5 was a progression of 3 different eras.
>
> I can't recall ever hearing the term 'quasi-quote' or
> 'quasi-expression'.  Macros and backquotes were just part of the Lisp
> language.
>
> When I got to C is when I got confused.  It seemed like the language
> wasn't finished.  Here's what happened when I first start using it:
>
> Me: "What is all this malloc / free nonsense?  I've never had to do this
> before."
> Coworker (CW): "What do you mean?  All languages require memory management."
> Me: "I haven't done this in 15 years of programming. Ever."
>
> Me: "Ok, show me how to figure out what is wrong."
> CW: "Fire up the debugger."
> Me: "Good, I know what that is...   Hmm, what's the value of this thing
> all in caps?"
> CW: "It can't tell you, that's a macro.  It's not part of the program."
> Me: "??  How can that be a macro if it never executes?  It sounds like
> an editor expansion that somehow escaped from from its environment."
> Me: "Ok, let me create a new variable and evaluate the expression it
> represents."
> CW: "You can't create a new variable silly."
> Me: "?? Ok, step into the function.  Wait, I see... the incoming arg is
> wrong.  Let's step back and change it."
> CW: "No can do."
> Me: "??  Ok, mentally I'll assume it is changed and...  oh I see the
> problem.  Let's change this function and continue!"
> CW: "What planet did you come from?  This is just a debugger!"
> Me: "Isn't a debugger the thing you use to write code?  You type it in
> to the listener and you modify it in the debugger.  When you're done you
> dump it in an editor using the pretty printer.  How do you guys write code?"
>
> And that was 15 years ago.  Now it is possible to use Visual programming
> tools... although you still can't program in the debugger.
>
>
> To me metaprogramming is writing code that dynamically changes the rules
> of the language.  Writing source code and calling the compiler isn't
> even generative programming because it feels like using a tool chain to
> script a series of things a human does at the keyboard.  Generating code
> (typically attributes, like when receiving a new XML stream with new
> attributes) that was not there when the compiler first ran is standard
> fare for a dynamic, runtime language.  Although I've not yet seen one
> code example in a commercial environment with XML where the
> implementation has to be recompiled to accept the new attributes.  Why?
> Because they use static languages and adding a new atom or string is
> runtime wizardry equated with automatic programming.
>
> If you want to see how metaprogramming really works, read "Art of the
> MetaObject Protocol" by Gregor Kiczales and Jim de Rivieres.  This book
> is as eye opening as reading a first book on how neutron stars become
> black holes.  It will either blow your mind or prevent you from having
> fun writing normal programs again.  Every programmer should read this book.
>
> [For those without the time, the book explains how OO is implemented in
> Lisp using StandardObject, StandardClass and StandardMethod and then
> shows how you can subclass those instances to **change the behavior of
> the object system** for example using a different implementation for
> storing a class or changing the semantics of inheritance or invoking a
> method.]
>
> On further reflection, the metaprogramming in the book is modifying the
> object-oriented system not Lisp itself.  At some point all things bottom
> out or else there are no assumptions the system makes.  The reason Lisp
> was so great was because data and code were one (in the greatest zen
> sense) so that you could manipulate code with the same ease as data.
> You didn't need to use a different language or API and then call a
> compiler and some other arcane incantations.
>
>
> Just some ramblings.  Not to detract from smerl which may prove to be a
> useful tool for erlang. Judging from wikipedia I am definitely in the
> minority on the classification of metaprogramming.
>
> It's just that erlang is structured with a VM and internal
> representations and weak text-substitution macros, so this area of
> research has rather limited potential IMHO.
>
> jay
>
>