Never trust a statistics you didn't forge yourself

[Ulf Wiger (AL/EAB)]

Michael Suess wrote:
> 
> I have thought long about whether or not I should try to 
> defend our study here, especially since we seem to have 
> raised a lot of hostility here for some reason.

I wouldn't say that. At least the first two comments 
on the study had smileys in them. (:

Personally, I can't recall a study where Erlang came
out more favourably, after all. Erlang even received 
special mention.

> You are of course free to feel that way. I knew that
> I would upset some people with this study, especially
> the ones whose "pet system" has not gotten many votes.
> What surprises me though is, that I the only flames I
> got until now come from the Erlang-community, although
> Erlang has gotten way more than its share of publicity
> in our paper. 

Why more than its share? Erlang clearly dominated the
"functional languages" category. This was worthy of 
a comment. 38 responses is not that much, either from
a statistical point of view, or considering that we
log about 10-15,000 downloads/month from the erlang.org
site. Clearly, most erlang users did _not_ respond
(if they had, you'd really have some explaining to 
do in your report ;-)


> I have paid careful attention to make it very 
> clear, how the study was carried out and whom
> I have contacted.

I agree.

> Fact is, that Joe posted about the survey on this mailing 
> list, you can still find his post in the archives:
> http://www.erlang.org/ml-archive/erlang-questions/200510/msg00171.html
> 
> Do any of you disagree, that Erlang has gotten a higher 
> number of votes because of this? I do not think so.

Of course, Joe's mail did very little beyond 
simply forwarding the link:

"If you use Erlang, why not tell

http://www.plm.eecs.uni-kassel.de/parasurvey/

About it

/Joe"

Most likely, Erlang would have received very few votes
wihout it, mainly because most erlang users would
not have known about the survey at all.

To compare, you offered $50 book gift certificates
in your own posts to e.g.
http://softwareforums.intel.com/ids/board/message?board.id=42&message.id
=676
http://softwareforums.intel.com/ids/board/message?board.id=42&message.id
=768
http://www.lam-mpi.org/MailArchives/lam/2005/10/11389.php

That is, you posted reminders as well in the other
forums, again reminding of gift certificates. Nothing
wrong with that -- you probably have to tease people
with some rewards to get them to respond -- but it 
does make Joe's "rallying cry" to the Erlang 
community rather modest.  (:


> I have an explanation for the relatively high number
> of votes for Erlang and I have been open about all
> other aspects of this survey. Why the heck should
> I keep this one secret??

As you mention in your report, Erlang is one of very
few languages with built in support for concurrency.
Practically all Erlang programs have some degree of
concurrency in them, and many who learn Erlang
will hesitate to use anything else for programming
parallel applications(*). I still think that the number
38 could have been significantly higher, esp if there
had been a reminder on the Erlang list as well, 
offering gift certificates. The number of Erlang
users who responded is only surprising given that
Erlang was not targeted for the survey. Like Luke,
I have no problem with your decision not to include
Erlang in the first place.

(*) Although the applications where Erlang is used 
differs somewhat from the ones you were most interested
in, as you've stated yourself. I venture to guess that
this is also true for e.g. the OCaml crowd (was it
the MLDonkey team that responded?  ;-)


> I have not been able to find any comparable data 
> anywhere at all.

Neither have I.

> I did the best I could to provide at least 
> some clues.

So you did. Thank you for that.

> None of the other systems was contacted by us, and
> as far as we know, Erlang was the only "Other"
> system, where the survey got noticed at all. 
> This is why the numbers for Erlang are unusually
> high and cannot be directly compared to the other
> systems, and this is why we mention it in the paper.
> Nothing more, nothing less. 

Fair enough. But in our experience, most users of 
functional languages in general do not deal with
concurrency much (this becomes obvious in forums
like comp.lang.functional), whereas practically
every erlang programmer around could have responded
to your survey that they often or always use Erlang
for programming parallel systems.


> When I asked for review on this paper, the Erlang
> numbers were held against me again and again, and
> it was even suggested to take them out.

Now, would you believe me when I say that this doesn't
surprise me in the least. (:

Last fall, there was a fairly lengthy thread on this
list about a small company debating a mix of niche
languages vs. C++. Here's one post that was 
particularly telling:
http://article.gmane.org/gmane.comp.lang.erlang.general/11706

There are many members on this list who have taken
a fair amount of heat in their time for favouring
something so decidedly "not-C++-or-Java" as Erlang.
Perhaps that makes us a bit too eager sometimes...

> I have decided not to do that, because I did not want
> to let my or anyone elses subjective opinion influence
> the results any more than necessary. Maybe I should
> have just listened... 

No, you did the right thing. (:


Regards,
Ulf Wiger