Never trust a statistics you didn't forge yourself

[Michael Suess]

Hi Joe, hi members of this list,

it is obvious by now, that there is not much I can do to convince you and at 
least some members of this list of the value of our study. Thats fine, and I 
can live with that. Just drop me a note, if you do not want to be notified of 
the results of our followup paper, and I will take you off my list.

However, some claims have been made here regarding the process I used to carry 
out and evaluate this study and about my scientific integrity, which I 
believe to be false. I will at least try to address them in this mail, along 
with a few more perspectives from my side.

On Wednesday 22 February 2006 13:09, Joe Armstrong (AL/EAB) wrote:
[snip]
> And what did I say here? -
>
> The complete post was:
>
>               If you use Erlang, why not tell
>
>                    http://www.plm.eecs.uni-kassel.de/parasurvey/
>
>               About it
>
>               /Joe
>
> Just compare this for a moment to what was posted to
> the LAM/MPI General User's Mailing List
>
> http://www.lam-mpi.org/MailArchives/lam/2005/10/11389.php
>
> I quote:
> 	> More than 50 people have filled out the survey this far, and
>
> therefore I will
>
>       > be evaluating the results shortly (it will close in two weeks,
>
> on November
>
>       > the 5th!). But before I do, please consider filling out the
>
> survey to make
>
>       > the results even more valuable. Of course, I will make the
>
> results available
>
>       > to everyone who participated. And before I forget to mention it:
>
> two gift
>
>       > certificates from amazon.com are being awarded to everyone who
>
> participated.
>
>       > Thank you for your cooperation,
>       > Best Regards,
>       > Michael Suess
>
> The bit about the gift certificates goes totally unmentioned in your
> paper.

p.2, Survey Methodology:
"If participants submitted their answers along with a working
e-mail address, they could win one of two 50$ gift certificates from 
amazon.com."


> You say  "An influential member of the Erlang community
> requested members of their mailing list to show there support of Erlang
> by participating
> in the Survey" - which is a false claim - I never said anything about
> support - I asked the
> people on this list to tell you about their experiences.

This seems to be the single most important point, that most people here are 
complaining about, right? Reading through your mail again, I admit that the 
phrase I used in the paper is quite strong and I will make sure to change it 
in my followup paper. I will even go as far as to say: 

"The phrase is misleading, my mistake, sorry about it."

I would be very interested in how you or all the other people on this list who 
have been bothered by it would phrase it, though, as I cannot leave out the 
fact that you posted on this list entirely. Without this, figure 4 gives a 
very wrong impression, as the Erlang people were the only ones (of the 
systems mentioned in this figure) notified of the survey (whether by you, or 
by me does not make a difference here, as others have claimed). 

> And then you omit to say that gift certificates were offered to members
> of other mailing
> lists, and finally when you get an unexpectly positive response from the
> members of the Erlang
> mailing list you dismiss this this result since I asked members of this
> list to
> participate in your survey.
>
> Usually in an academic paper it is consider de rigour to describe your
> experimental procedure.
> Omitting to mention that you offered gift certificates to people who
> filled in the
> survey is rather strange since it is probably that it will bias the
> results.
>
> Note that doing so you may well have biased the results in favour of MPI
> - MPI got the highest
> rating among parallel programming systems - was this because you offered
> them gift certificates?
>
> If you are going to make unsubstantiated claims in your paper about the
> supposed
> influence of any mailing that I made to this list - then you should also
> mention the
> ways in which you other results might be biased.
>
> And by the way - you haven't sent me a gift certificate - was the offer
> only open
> to people on the MPI list? - did the other people on this list get any
> gift certificates?

This paragraph made me very angry at first, then very sad. I do not know how 
long it took you to write it, but it took me about 10 seconds to open the 
report, press "find" and search for "gift". There is only one result, and 
this is the quote given above. And it is exactly where it is supposed to be, 
right under "Survey Methodology". Is it really too much to ask of you to do 
the same and actually invest these 10 seconds, before you write 7 paragraphs 
of false claims about my scientific methodology and try to attack it on this 
level?

You also alledge that I have skewed the results in favour of MPI by offering 
them gift certificates. Had you taken the time to check the other mailing 
lists and forums, you would have noticed that I send all of them EXACTLY the 
same mail, and all with the same offer. An offer which is also right on top 
of the survey.

You will also notice, that the last sentence of each of these mails contains a 
mistake:
 "And before I forget to mention it: two gift 
certificates from amazon.com are being awarded to everyone who participated."

This should more accurately read "among everyone who participated". I have 
noticed this only now, and this is the second mistake I will admin in this 
post. To my defense I can say that the right phrase was used at the top of 
the survey, and I will assume that everyone read that before he filled out 
the survey.

Nevertheless, the winners of these certificates have been drawn, and their 
rewards have been given to them on January the 4th. One of them allowed me to 
publish his name, and it is Timothy Mattson of Intel fame. I have not gotten 
an answer from the other winner, about whether or not I can publish his name, 
and therefore I have to keep it secret. 

The last point I want to raise about these 7 paragraphs: you alledge that I do 
not mention the other ways in which my results could be biased. I can only 
kindly ask you to reread the section about "Survey Methodology", and you will 
find that I have made exactly this point very clear.

[snip]
> To start with Erlang is unique in your survey in the sense that
> concurrency is part of the language and not the OS.
>
> Let me give you some examples:
>
> 	c++ is a language but NOT a concurrent programming system
> 	PVM is a concurrent programming system but NOT a language
>
> This is true of ALL the languages/systems in your paper EXCEPT Erlang.
>
>       Erlang is a language AND a concurrent programming system.
>
>       And thus is belongs to both figures 2 and 5.

I thought I had made this clear when you first wrote me about it in private 
mail. In Figures 2 and 5 you will find only the languages I explicitly asked 
about in my survey, and not the languages people put in by hand in any of the 
"other" options. To put all in one graph would have been extremely unfair 
against the "other" languages, don't you think? If I had explicitly asked for 
Erlang (or for Python or Ocaml) in questions 1 or 2, they would have gotten a 
higher number of votes than they have now. Had I not asked for Java, it would 
have gotten fewer votes. And therefore I can only compare the languages I 
explicitly asked for with each other, and the "other" languages with each 
other. And this is what I do.

On the other hand, Erlang gets mentioned both for the "other languages" and 
for the "other systems" - only that there is no graph for the other systems, 
because only Erlang and .NET would show up in there at all.

Let me quote the report again:
"The other parallel programming systems submitted include a wide variety of 
systems, yet only Erlang (26 submissions, accumulating to a usage of 89) 
and .NET (four submissions with a usage of 10) managed to be mentioned more 
than three times. Noteworthy is the fact that Erlang is one of the very few 
programming languages for which parallelism is an integral part of the 
language, and it therefore has high submissions for both questions one
and two."

I know the point about Erlang being a language and a parallel programming 
system you are trying to make here, I have known it when I wrote the paper, I 
have known it when I first answered your private mail and I still know it 
now. What I do not know is what to do to make you understand, that I know it 
and that I am not here to bash Erlang or piss off you or the Erlang-community 
in any way!

>       Figure 2 is also incorrectly labelled - the caption is "Parallel
> programming languages"
> here you mention C, C++, Fortran, java , and something called functional
> and logical
>
>       This is very misleading functional and logical are NOT languages
> they are classes
> of languages.

Thanks for educating me again, Joe. If you look at the survey (or figure 1 for 
that matter), you will see that I know the difference. Yet I am sure you 
understand that I had space constraints when I put the labels on the graph...

>      C, Fortran are NOT parallel languages.

This is getting ridiculous. Please take a look at the survey again, or at 
figure 1 where the exact phrase of question one is printed:
"How often have you used the following programming languages as a basis for 
your parallel applications during the last 3 years?"

You will not argue with me, that these languages are base languages for many 
parallel programming systems, or will you? Could I have put that in the label 
of figure 2 again? Yes, but when I wrote the paper this did not occur to me, 
as the whole paper around the figures makes this point perfectly clear in my 
humble opinion.

[snip]
I am running out of time, and I cannot comment on the rest of the points made 
in your mail anyways, as I do not know the Erlang community enough. I will 
therefore try to address some other points made by other people:

Andrae Muys wrote:
> A truly representative survey would have identified a community of users who 
> write parallel programs, and have targeted them with solicitations

And this is exactly the reason, why I have claimed no statistical value in the 
data whatsoever. I even write in the report itself:

"For this reason, please take all results of this survey with a grain of salt, 
as they are not statistically sound! For statistical significance, we would 
have to sample a proportional part of the parallel programming population, 
and we know of no way to do so (at least not within
our budget). It is for this reason, that you will not find any statistical 
measures applied to the data in this paper."

And there you will also find the answer I would like to give to your claim. We 
know the data are not hard. And I have also done my homework and thought 
about how I could make these data more statistically useful, but I have come 
to the conclusion that I do not know how! And I am fairly sure when you go 
further than rough sketches of your plans to identify, sample and contact 
these subcommunities, you will come to a similar result, at least when you 
consider that this is just a side-track of my research and that we do not 
have as much money as we wish sometimes. But maybe you can prove me wrong and 
do a better survey, I would sure be interested in the results...

>This is appalling research, I am interested in knowing which journal 
>accepted this paper so I know which journal I can avoid because any 
>peer-review that passes this tautological waste of effort is not 
>peer-reviewed in any meaningful sense of the phrase. 

*sigh*. Thank you for the very constructive criticism.
Let me quote myself again:
>I will be working on an extended version of this paper including all results 
>and hope to publish it in a research journal soon (I have contacted some 
>journals, and although this is not exactly traditional research, they have 
>showed interest)

Note the part about the "extended version" and "showing interest". The 
extended version is not even written, nor peer-reviewed, all I did was ask 
some journals about whether or not they would accept a paper of this scope at 
all. 

Marc van Woerkom wrote:
>The issue here is not some ranking but scientific honesty.
>You have IMHO not dealt appropriately with the answers to 
>your survey. Other posts here described the flaws better 
>than I am able to do it.
I think, I have spent the better part of this evening to answer the main 
attempts to challenge my scientific honesty or integrity. I have admitted two 
mistakes, which are not related to my honesty or integrity in any way. I have 
not put any claims in the paper I am not ready to defend here or anywhere 
else (with the possible exception noted at the beginning of this mail). I am 
the first to admit that I do make mistakes, but I am also getting tired of 
rehashing what I said in the paper or to fight things I am supposed to have 
done or not done, many of which are false or quoted out of context.

>As an example that bad rankings don't cause any stirr take 
>e.g. the big language shootout, which is discussed on this 
>list occassionaly.

And the reason for this could be, that they do not comment on the results at 
all ? 

Ulf Wiger wrote:
>Why more than its share? Erlang clearly dominated the
>"functional languages" category. This was worthy of 
>a comment.

Yes, and I have made that comment, and a few more about Erlang. Just search 
for Erlang in the paper, and you will find many results. And thats what I 
mean by "its share of publicity". 

[snip]
>Most likely, Erlang would have received very few votes
>wihout it, mainly because most erlang users would
>not have known about the survey at all.
I have been trying to make that point somewhere at the top of this mail. Nice 
to see I am not alone in this regard.

>That is, you posted reminders as well in the other
>forums, again reminding of gift certificates. Nothing
>wrong with that -- you probably have to tease people
>with some rewards to get them to respond -- but it 
>does make Joe's "rallying cry" to the Erlang 
>community rather modest.  (:
I know that now, and thats why I admitted that mistake.

[snip]
>Although the applications where Erlang is used 
>differs somewhat from the ones you were most interested
>in, as you've stated yourself. I venture to guess that
>this is also true for e.g. the OCaml crowd (was it
>the MLDonkey team that responded?  ;-)

I do not know, who responded in favour of OCaml, but I might add that I have 
looked at Ocaml very closely at the beginning of my research. Unfortunately, 
it has the same "problem" as Erlang: no speedups for parallel programs. And 
given my background and my goals, this is a showstopper. 

>> When I asked for review on this paper, the Erlang
>> numbers were held against me again and again, and
>> it was even suggested to take them out.

>Now, would you believe me when I say that this doesn't
>surprise me in the least. (:

>Last fall, there was a fairly lengthy thread on this
>list about a small company debating a mix of niche
>languages vs. C++. Here's one post that was 
>particularly telling:
>http://article.gmane.org/gmane.comp.lang.erlang.general/11706

>There are many members on this list who have taken
>a fair amount of heat in their time for favouring
>something so decidedly "not-C++-or-Java" as Erlang.
>Perhaps that makes us a bit too eager sometimes...

I don't know.

What I also do not know, is how to end this mail.

Therefore, I will just say thank you for listening,
best regards,
Michael

-- 
"What we do in life, echos in eternity..."
M.: msuess@REDACTED | T.: +49-561-804-6269 | F.: +49-561-804-6219
WWW: http://www.plm.eecs.uni-kassel.de/plm/index.php?id=msuess
Public PGP key and fingerprint available at above address.
Research Associate, Programming Languages / Methodologies Research Group
University of Kassel, Wilhelmshöher Allee 73, D-34121 Kassel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20060223/fa158365/attachment.bin>