[erlang-questions] Erlang web servers challenge

[Richard O'Keefe]

On 9/07/2011, at 2:44 AM, Ulf Wiger wrote:
> Having said this, I invite anyone who goes through that kind of exercise to share their results. Not only will it help you evaluate the experiment honestly; it will increase the store of experiments that can be copied and tailored to the specific challenges of the next project.

I had an unpleasant experimental experience of my own last year.
Let me first give you the lesson I learned, and then the background.

LESSON: Expect your experiment to surprise you,
	probably by showing the experiment was a waste of time.

Background: I'm sick of arguments about style.  To my mind, it is so
obvious that baStudlyCaps isAVeryStupidWayToWriteIndeed and I
cannotUnderstandWhyOtherPeopleDoNotSeeThat.  But they don't.  My zeroth,
the New Zealand Anglican Church has even brought out an electronic
version of the liturgy called WePray, in a desperate attempt to seem
hip.  (Since it is only available for an operating system sold by what
may be the largest software company to have been convicted to software
piracy, I wonder what their ethics committee were doing.  But I digress.)

So I devised a little language called Chatterton
(http://www.cs.otago.ac.nz/cosc345/chatterton.pdf),
which allowed me to mechanically produce several style variants of
some sample programs and ask some 3rd year software engineering students
to find some mistakes in them.

What I expected was one of three things:
 - no measurable effect
 - more readable code (i.e., NOT baStudlyCaps) being easier to fix
 - more familiar (i.e., JustLikeXingJava) being easier to fix.

What I *got* was students telling me they couldn't read code on
paper; they needed syntax-colouring IDEs (the listings all fitted
on a single sheet of paper and used black-and-white styling) and
ideally a debugger so they could find mistakes by stepping through
the code.  I also got students telling me that it was horribly
unreasonable of me to expect them to read a 30-page manual; NOBODY
could read that much.  And finding a definition of an identifier
in a 2-page listing is just beyond human capacity; it's impossible
to do that without the machine assistance of an IDE.

So the whole experiment produced no worthwhile data for reasons having
nothing to do with what I thought I was testing.

As other people have been saying, the "Erlang web servers challenge"
is at serious risk of producing no worthwhile data.