[erlang-questions] Erlang 3000?

Joe Armstrong erlang@REDACTED
Tue Nov 18 09:58:56 CET 2008 

let's throw in my 5 cents ...

It is (of course) highly desirable to have a large set of libraries
that are written in a consistent and predictable manner - the problem
is how to we get there?

I have actually been building a library that one day I hope to
publish - it's got 70 modules in it so far ...

Now there are some problems in building a large library.

For reasons of consistency all the library modules have to follow
some common programming rules - these rules have to encapsulate what
we think is good style and best practise (whatever that means)

Let's now imagine what happens when you build a large library.

    -  you write the first module - no problems
    -  you write the second module - ...
    -  ...
    -  you write the N'th module now you realize the design of
       some functions you want to call in module 6 was wrong.
       What do you do? - stop writing module N and go back to 6.
       In re-writing 6 you discover that 2 was wrong, what do you
       do? stop writing module 6 and go back to 2 ????

       This is bad enough when one person writes all the code
       but when somebody else wrote the earlier code the problem
       gets *much* worse.

    - you've been a bit lazy with the documentation so you decide to
      write some decent documentation. When you get started you realise
      that what you are describing is a mess - it would be much easier
      if it had been written "like this" so you write the documentation
      describing what the code should have been - then go back and
      hack the code.

    - You've written the code but not much meta data. The meta data
      is the stuff describing the code (the type signatures, documentation
      etc.)

    - when the code base is small there is no problem (put it all in one
      directory) - as it grows you wish you'd organised it better
      so you put it in several directories. As it gets very large you
      realise that putting it into several directories makes doesn't
      help - there are so many directories that you can't find the code
      anyway - the only way to find things is to hope that Google has
      indexed it.

     - you wish you had added more meta data to your code (to allow
       accurate indexing)

     *hypothesis* as code bases get larger the most important thing
     is the quality of the meta data - not the data. The meta data
     determines our ability to find stuff - if we can't find stuff
     it does not matter how good it is.

So how do I organise my 70 module library?

      - all code in one directory
      - whenever I find an inconsistency in an earlier module I go
        back and rewrite the original code (and everything that calls it)

One thing I find *very* useful is to put all small pure functions
that are generally useful in the *same* module.

<< There is a problem - functions have to belong to modules - but there are a
*lot* of very small pure functions that can't really be said to belong to
any particular module solution: Put them all in lib_extra.>>

<<
I also use inheritance (If I don't want to touch an existing module) I
write.

     -module(xx).
     -inherit(old_mod).
     -export([....])

This is just a simple parse transform, it makes stub functions for all the
exported functions in old_mod that are not in xx. if xx has a new version
of a routine in old_mod it will override that function.

Why do it like this? - because I may not want to modify old_mod (if I have
distributed it).
>>

At the end of the day I believe the meta data to be far more important
than the code. getting the APIs right is the difficult bit.

If anybody else tells me "I have to read the code to see what it does"
I will scream - whenever I read code the following happens:

     - I don't understand it
     - I start to understand it
     - I get an overwhelming desire to rewrite it

Reading code to discover what it does is a sign of incompleteness. A
program is not complete until it has passed all its tests and has good
documentation.

If you have to read the code to figure out what the program does then the
project is incomplete.

We should really have a programming rule. The APIs and code should be
in two *different* files - the reviewers should not be allowed to read
the code (This is a *very good reason* why code and documentation
should not be allowed in the same file)

(( If code and documentation are allowed in the same file, then the
person editing the documentation might accidentally read the code and
thus describe the code *and not the interface* (which is a completely
different thing))

Flame on: reading the code tells you what the code does (if you can
understand it - but not "what it is *supposed* to do".

So before we rush off and write the ultimate set of modules - can we
try to write just ONE module in as beautiful way as possible.

This module could serve as the great progenitor - it should be
well written, well documented, with well thought-out meta data.

*I know of no such module* (and in case you were wondering, my 70 module
library is *not* beautifully annotated with meta-data).

There is also the problem of style and taste - personally I favor the
"one author many critical eyes" approach. When my Erlang book went to
beta I got thousands of eyes staring at every line I'd written - this
feedback was fantastic (thanks everybody) and significantly improved
the quality.

I don't believe in a group of people "voting" for features in code -
But I do believe in readers and writers. Few writers of code (who make the
decisions) but many readers.

If we set this thong up I will donate parts of my library to the
project - I've written a new XML thing that is rather pretty (I think)
- it lacks test cases (because it's right :-) and meta data (because
I'm lazy) and documentation (because we don't have a good
documentation system in place)

I think it would be great to set up an infrastructure for writing good
quality community code - the things we have to setup are:

   - programming rules
   - review process
   - distribution process
   - production process

We need methods for producing drop-dead-beautiful documentation
(in PDF, HTML, ...) - think "book quality"

We need an agreement on meta data.

We need to an agreement on the distribution process.

We need an infrastructure. The genius of the Wikipedia is that it is
an organisational form that allows a lot of people to contribute by
doing a little. Correcting a spelling mistake, asking a question,
replying to a question has enormous power when done by hundreds or
thousands of people.

So here we have it - lots of jobs to be done.

Me, I can author code, if somebody else can deign and host a web site
with SW support for the review process it would be great.

/Joe Armstrong




On Tue, Nov 18, 2008 at 8:05 AM, Bengt Kleberg
<bengt.kleberg@REDACTED> wrote:
> Greetings,
>
> One good (IMHO) critique of packages, as they are currently implemented,
> is found here:
> http://www.erlang.org/pipermail/erlang-questions/2006-November/023782.html
>
> On the other hand, packages, even as they are currently implemented,
> would make it possible to rewrite the libraries. That could also be done
> with the '.' replaced with '_' so it is not a very good argument. But I
> really would like to rewrite them.
>
>
> bengt
> Those were the days...
>    EPO guidelines 1978: "If the contribution to the known art resides
>    solely in a computer program then the subject matter is not
>    patentable in whatever manner it may be presented in the claims."
>
>
> On 2008-11-18 03:25, Dave Smith wrote:
>>
>> So, what's the status of packages?
>>
>> Robert, Is you're objection to this particular implementation or is
>> there some argument against an hierarchical module namespace in general?
>>
>> 2008/11/17 Vance Shipley <vances@REDACTED <mailto:vances@REDACTED>>
>>
>>     On Tue, Nov 18, 2008 at 02:10:19AM +0100, Robert Virding wrote:
>>     }  No way, don't like packages.
>>
>>     Well somebody around here was working on that at some time.
>>
>>            -Vance
>>
>>     To be clear:  http://erlang.org/doc/man/packages.html
>>     _______________________________________________
>>     erlang-questions mailing list
>>     erlang-questions@REDACTED <mailto:erlang-questions@REDACTED>
>>     http://www.erlang.org/mailman/listinfo/erlang-questions
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://www.erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>

More information about the erlang-questions mailing list