Never trust a statistics you didn't forge yourself
Thu Feb 23 00:55:02 CET 2006
Hi Joe, hi members of this list,
it is obvious by now, that there is not much I can do to convince you and at
least some members of this list of the value of our study. Thats fine, and I
can live with that. Just drop me a note, if you do not want to be notified of
the results of our followup paper, and I will take you off my list.
However, some claims have been made here regarding the process I used to carry
out and evaluate this study and about my scientific integrity, which I
believe to be false. I will at least try to address them in this mail, along
with a few more perspectives from my side.
On Wednesday 22 February 2006 13:09, Joe Armstrong (AL/EAB) wrote:
> And what did I say here? -
> The complete post was:
> If you use Erlang, why not tell
> About it
> Just compare this for a moment to what was posted to
> the LAM/MPI General User's Mailing List
> I quote:
> > More than 50 people have filled out the survey this far, and
> therefore I will
> > be evaluating the results shortly (it will close in two weeks,
> on November
> > the 5th!). But before I do, please consider filling out the
> survey to make
> > the results even more valuable. Of course, I will make the
> results available
> > to everyone who participated. And before I forget to mention it:
> two gift
> > certificates from amazon.com are being awarded to everyone who
> > Thank you for your cooperation,
> > Best Regards,
> > Michael Suess
> The bit about the gift certificates goes totally unmentioned in your
p.2, Survey Methodology:
"If participants submitted their answers along with a working
e-mail address, they could win one of two 50$ gift certiﬁcates from
> You say "An influential member of the Erlang community
> requested members of their mailing list to show there support of Erlang
> by participating
> in the Survey" - which is a false claim - I never said anything about
> support - I asked the
> people on this list to tell you about their experiences.
This seems to be the single most important point, that most people here are
complaining about, right? Reading through your mail again, I admit that the
phrase I used in the paper is quite strong and I will make sure to change it
in my followup paper. I will even go as far as to say:
"The phrase is misleading, my mistake, sorry about it."
I would be very interested in how you or all the other people on this list who
have been bothered by it would phrase it, though, as I cannot leave out the
fact that you posted on this list entirely. Without this, figure 4 gives a
very wrong impression, as the Erlang people were the only ones (of the
systems mentioned in this figure) notified of the survey (whether by you, or
by me does not make a difference here, as others have claimed).
> And then you omit to say that gift certificates were offered to members
> of other mailing
> lists, and finally when you get an unexpectly positive response from the
> members of the Erlang
> mailing list you dismiss this this result since I asked members of this
> list to
> participate in your survey.
> Usually in an academic paper it is consider de rigour to describe your
> experimental procedure.
> Omitting to mention that you offered gift certificates to people who
> filled in the
> survey is rather strange since it is probably that it will bias the
> Note that doing so you may well have biased the results in favour of MPI
> - MPI got the highest
> rating among parallel programming systems - was this because you offered
> them gift certificates?
> If you are going to make unsubstantiated claims in your paper about the
> influence of any mailing that I made to this list - then you should also
> mention the
> ways in which you other results might be biased.
> And by the way - you haven't sent me a gift certificate - was the offer
> only open
> to people on the MPI list? - did the other people on this list get any
> gift certificates?
This paragraph made me very angry at first, then very sad. I do not know how
long it took you to write it, but it took me about 10 seconds to open the
report, press "find" and search for "gift". There is only one result, and
this is the quote given above. And it is exactly where it is supposed to be,
right under "Survey Methodology". Is it really too much to ask of you to do
the same and actually invest these 10 seconds, before you write 7 paragraphs
of false claims about my scientific methodology and try to attack it on this
You also alledge that I have skewed the results in favour of MPI by offering
them gift certificates. Had you taken the time to check the other mailing
lists and forums, you would have noticed that I send all of them EXACTLY the
same mail, and all with the same offer. An offer which is also right on top
of the survey.
You will also notice, that the last sentence of each of these mails contains a
"And before I forget to mention it: two gift
certificates from amazon.com are being awarded to everyone who participated."
This should more accurately read "among everyone who participated". I have
noticed this only now, and this is the second mistake I will admin in this
post. To my defense I can say that the right phrase was used at the top of
the survey, and I will assume that everyone read that before he filled out
Nevertheless, the winners of these certificates have been drawn, and their
rewards have been given to them on January the 4th. One of them allowed me to
publish his name, and it is Timothy Mattson of Intel fame. I have not gotten
an answer from the other winner, about whether or not I can publish his name,
and therefore I have to keep it secret.
The last point I want to raise about these 7 paragraphs: you alledge that I do
not mention the other ways in which my results could be biased. I can only
kindly ask you to reread the section about "Survey Methodology", and you will
find that I have made exactly this point very clear.
> To start with Erlang is unique in your survey in the sense that
> concurrency is part of the language and not the OS.
> Let me give you some examples:
> c++ is a language but NOT a concurrent programming system
> PVM is a concurrent programming system but NOT a language
> This is true of ALL the languages/systems in your paper EXCEPT Erlang.
> Erlang is a language AND a concurrent programming system.
> And thus is belongs to both figures 2 and 5.
I thought I had made this clear when you first wrote me about it in private
mail. In Figures 2 and 5 you will find only the languages I explicitly asked
about in my survey, and not the languages people put in by hand in any of the
"other" options. To put all in one graph would have been extremely unfair
against the "other" languages, don't you think? If I had explicitly asked for
Erlang (or for Python or Ocaml) in questions 1 or 2, they would have gotten a
higher number of votes than they have now. Had I not asked for Java, it would
have gotten fewer votes. And therefore I can only compare the languages I
explicitly asked for with each other, and the "other" languages with each
other. And this is what I do.
On the other hand, Erlang gets mentioned both for the "other languages" and
for the "other systems" - only that there is no graph for the other systems,
because only Erlang and .NET would show up in there at all.
Let me quote the report again:
"The other parallel programming systems submitted include a wide variety of
systems, yet only Erlang (26 submissions, accumulating to a usage of 89)
and .NET (four submissions with a usage of 10) managed to be mentioned more
than three times. Noteworthy is the fact that Erlang is one of the very few
programming languages for which parallelism is an integral part of the
language, and it therefore has high submissions for both questions one
I know the point about Erlang being a language and a parallel programming
system you are trying to make here, I have known it when I wrote the paper, I
have known it when I first answered your private mail and I still know it
now. What I do not know is what to do to make you understand, that I know it
and that I am not here to bash Erlang or piss off you or the Erlang-community
in any way!
> Figure 2 is also incorrectly labelled - the caption is "Parallel
> programming languages"
> here you mention C, C++, Fortran, java , and something called functional
> and logical
> This is very misleading functional and logical are NOT languages
> they are classes
> of languages.
Thanks for educating me again, Joe. If you look at the survey (or figure 1 for
that matter), you will see that I know the difference. Yet I am sure you
understand that I had space constraints when I put the labels on the graph...
> C, Fortran are NOT parallel languages.
This is getting ridiculous. Please take a look at the survey again, or at
figure 1 where the exact phrase of question one is printed:
"How often have you used the following programming languages as a basis for
your parallel applications during the last 3 years?"
You will not argue with me, that these languages are base languages for many
parallel programming systems, or will you? Could I have put that in the label
of figure 2 again? Yes, but when I wrote the paper this did not occur to me,
as the whole paper around the figures makes this point perfectly clear in my
I am running out of time, and I cannot comment on the rest of the points made
in your mail anyways, as I do not know the Erlang community enough. I will
therefore try to address some other points made by other people:
Andrae Muys wrote:
> A truly representative survey would have identified a community of users who
> write parallel programs, and have targeted them with solicitations
And this is exactly the reason, why I have claimed no statistical value in the
data whatsoever. I even write in the report itself:
"For this reason, please take all results of this survey with a grain of salt,
as they are not statistically sound! For statistical signiﬁcance, we would
have to sample a proportional part of the parallel programming population,
and we know of no way to do so (at least not within
our budget). It is for this reason, that you will not ﬁnd any statistical
measures applied to the data in this paper."
And there you will also find the answer I would like to give to your claim. We
know the data are not hard. And I have also done my homework and thought
about how I could make these data more statistically useful, but I have come
to the conclusion that I do not know how! And I am fairly sure when you go
further than rough sketches of your plans to identify, sample and contact
these subcommunities, you will come to a similar result, at least when you
consider that this is just a side-track of my research and that we do not
have as much money as we wish sometimes. But maybe you can prove me wrong and
do a better survey, I would sure be interested in the results...
>This is appalling research, I am interested in knowing which journal
>accepted this paper so I know which journal I can avoid because any
>peer-review that passes this tautological waste of effort is not
>peer-reviewed in any meaningful sense of the phrase.
*sigh*. Thank you for the very constructive criticism.
Let me quote myself again:
>I will be working on an extended version of this paper including all results
>and hope to publish it in a research journal soon (I have contacted some
>journals, and although this is not exactly traditional research, they have
Note the part about the "extended version" and "showing interest". The
extended version is not even written, nor peer-reviewed, all I did was ask
some journals about whether or not they would accept a paper of this scope at
Marc van Woerkom wrote:
>The issue here is not some ranking but scientific honesty.
>You have IMHO not dealt appropriately with the answers to
>your survey. Other posts here described the flaws better
>than I am able to do it.
I think, I have spent the better part of this evening to answer the main
attempts to challenge my scientific honesty or integrity. I have admitted two
mistakes, which are not related to my honesty or integrity in any way. I have
not put any claims in the paper I am not ready to defend here or anywhere
else (with the possible exception noted at the beginning of this mail). I am
the first to admit that I do make mistakes, but I am also getting tired of
rehashing what I said in the paper or to fight things I am supposed to have
done or not done, many of which are false or quoted out of context.
>As an example that bad rankings don't cause any stirr take
>e.g. the big language shootout, which is discussed on this
And the reason for this could be, that they do not comment on the results at
Ulf Wiger wrote:
>Why more than its share? Erlang clearly dominated the
>"functional languages" category. This was worthy of
Yes, and I have made that comment, and a few more about Erlang. Just search
for Erlang in the paper, and you will find many results. And thats what I
mean by "its share of publicity".
>Most likely, Erlang would have received very few votes
>wihout it, mainly because most erlang users would
>not have known about the survey at all.
I have been trying to make that point somewhere at the top of this mail. Nice
to see I am not alone in this regard.
>That is, you posted reminders as well in the other
>forums, again reminding of gift certificates. Nothing
>wrong with that -- you probably have to tease people
>with some rewards to get them to respond -- but it
>does make Joe's "rallying cry" to the Erlang
>community rather modest. (:
I know that now, and thats why I admitted that mistake.
>Although the applications where Erlang is used
>differs somewhat from the ones you were most interested
>in, as you've stated yourself. I venture to guess that
>this is also true for e.g. the OCaml crowd (was it
>the MLDonkey team that responded? ;-)
I do not know, who responded in favour of OCaml, but I might add that I have
looked at Ocaml very closely at the beginning of my research. Unfortunately,
it has the same "problem" as Erlang: no speedups for parallel programs. And
given my background and my goals, this is a showstopper.
>> When I asked for review on this paper, the Erlang
>> numbers were held against me again and again, and
>> it was even suggested to take them out.
>Now, would you believe me when I say that this doesn't
>surprise me in the least. (:
>Last fall, there was a fairly lengthy thread on this
>list about a small company debating a mix of niche
>languages vs. C++. Here's one post that was
>There are many members on this list who have taken
>a fair amount of heat in their time for favouring
>something so decidedly "not-C++-or-Java" as Erlang.
>Perhaps that makes us a bit too eager sometimes...
I don't know.
What I also do not know, is how to end this mail.
Therefore, I will just say thank you for listening,
"What we do in life, echos in eternity..."
M.: | T.: +49-561-804-6269 | F.: +49-561-804-6219
Public PGP key and fingerprint available at above address.
Research Associate, Programming Languages / Methodologies Research Group
University of Kassel, Wilhelmshöher Allee 73, D-34121 Kassel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: not available
More information about the erlang-questions