Never trust a statistics you didn't forge yourself
Joe Armstrong (AL/EAB)
Thu Feb 23 12:41:58 CET 2006
> -----Original Message-----
> From: Michael Suess [mailto:msuess@REDACTED]
> Sent: den 23 februari 2006 00:55
> To: Joe Armstrong (AL/EAB)
> Cc: Marc van Woerkom; leopold@REDACTED;
> Subject: Re: Never trust a statistics you didn't forge yourself
> Hi Joe, hi members of this list,
> it is obvious by now, that there is not much I can do to
> convince you and at least some members of this list of the
> value of our study. Thats fine, and I can live with that.
> Just drop me a note, if you do not want to be notified of the
> results of our followup paper, and I will take you off my list.
> However, some claims have been made here regarding the
> process I used to carry out and evaluate this study and about
> my scientific integrity, which I believe to be false. I will
> at least try to address them in this mail, along with a few
> more perspectives from my side.
> On Wednesday 22 February 2006 13:09, Joe Armstrong (AL/EAB) wrote:
> > And what did I say here? -
> > The complete post was:
> > If you use Erlang, why not tell
> > http://www.plm.eecs.uni-kassel.de/parasurvey/
> > About it
> > /Joe
> > Just compare this for a moment to what was posted to the LAM/MPI
> > General User's Mailing List
> > http://www.lam-mpi.org/MailArchives/lam/2005/10/11389.php
> > I quote:
> > > More than 50 people have filled out the survey this far, and
> > therefore I will
> > > be evaluating the results shortly (it will close in
> two weeks,
> > on November
> > > the 5th!). But before I do, please consider filling out the
> > survey to make
> > > the results even more valuable. Of course, I will make the
> > results available
> > > to everyone who participated. And before I forget
> to mention it:
> > two gift
> > > certificates from amazon.com are being awarded to
> everyone who
> > participated.
> > > Thank you for your cooperation,
> > > Best Regards,
> > > Michael Suess
> > The bit about the gift certificates goes totally
> unmentioned in your
> > paper.
> p.2, Survey Methodology:
> "If participants submitted their answers along with a working
> e-mail address, they could win one of two 50$ gift
> certiﬁcates from amazon.com."
> > You say "An influential member of the Erlang community
> > requested members of their mailing list to show there
> support of Erlang
> > by participating
> > in the Survey" - which is a false claim - I never said
> anything about
> > support - I asked the
> > people on this list to tell you about their experiences.
> This seems to be the single most important point, that most
> people here are
> complaining about, right? Reading through your mail again, I
> admit that the
> phrase I used in the paper is quite strong and I will make
> sure to change it
> in my followup paper. I will even go as far as to say:
> "The phrase is misleading, my mistake, sorry about it."
> I would be very interested in how you or all the other people
> on this list who
> have been bothered by it would phrase it, though, as I cannot
> leave out the
> fact that you posted on this list entirely. Without this,
> figure 4 gives a
> very wrong impression, as the Erlang people were the only
> ones (of the
> systems mentioned in this figure) notified of the survey
> (whether by you, or
> by me does not make a difference here, as others have claimed).
> > And then you omit to say that gift certificates were
> offered to members
> > of other mailing
> > lists, and finally when you get an unexpectly positive
> response from the
> > members of the Erlang
> > mailing list you dismiss this this result since I asked
> members of this
> > list to
> > participate in your survey.
> > Usually in an academic paper it is consider de rigour to
> describe your
> > experimental procedure.
> > Omitting to mention that you offered gift certificates to people who
> > filled in the
> > survey is rather strange since it is probably that it will bias the
> > results.
> > Note that doing so you may well have biased the results in
> favour of MPI
> > - MPI got the highest
> > rating among parallel programming systems - was this
> because you offered
> > them gift certificates?
> > If you are going to make unsubstantiated claims in your
> paper about the
> > supposed
> > influence of any mailing that I made to this list - then
> you should also
> > mention the
> > ways in which you other results might be biased.
> > And by the way - you haven't sent me a gift certificate -
> was the offer
> > only open
> > to people on the MPI list? - did the other people on this
> list get any
> > gift certificates?
> This paragraph made me very angry at first, then very sad. I
> do not know how
> long it took you to write it, but it took me about 10 seconds
> to open the
> report, press "find" and search for "gift". There is only one
> result, and
> this is the quote given above. And it is exactly where it is
> supposed to be,
> right under "Survey Methodology". Is it really too much to
> ask of you to do
> the same and actually invest these 10 seconds, before you
> write 7 paragraphs
> of false claims about my scientific methodology and try to
> attack it on this
You're right - and I missed this part of the paper - sorry.
When I wrote this I had just seen the posting to the net which said:
"two gift certificates from amazon.com are being awarded to everyone
The paper said
".. they could win one of two 50$ gift certificates"
These mean different things. You said that my posting might have biased the results - this annoyed me and a lot of people on the Erlang list
who responded to your survey, and I annoy you by pointing out that
you have not accurately described your methods.
It is not my intention to anger you, and I am sure that you did not
intend to annoy various members of the Erlang list - but it was my intention to point out the difference between facts and speculations
in your paper. It will be a better paper if you stick to the numbers,
and leave off any attempt at guessing as to why they are as they are.
I suggest you revise the paper a bit.
You could say:
- I sent the following post to [list them] newsgroups
" quote what you sent "
As a result of this a number of message asking people to participate in the survey were cross posted to a number of different groups.
It is impossible to estimate how the results of these messages biased
the responses from the different groups.
My personal opinions is that:
- Offers of gift certificate has no effect on people filling
in the survey, and
- My posting to the Erlang list has no effect (other than
making people aware that there was a survey - I think the
response from the Erlang list would have been the same
no matter who had sent the mail)
[can we put this bit of the argument to rest now and wave a few olive branches around - I have a program to write ]
More information about the erlang-questions