Never trust a statistics you didn't forge yourself

[Joe Armstrong (AL/EAB)]

> -----Original Message-----
> From: Michael Suess [mailto:msuess@REDACTED] 
> Sent: den 23 februari 2006 00:55
> To: Joe Armstrong (AL/EAB)
> Cc: Marc van Woerkom; leopold@REDACTED; 
> erlang-questions@REDACTED
> Subject: Re: Never trust a statistics you didn't forge yourself
> 
> Hi Joe, hi members of this list,
> 
> it is obvious by now, that there is not much I can do to 
> convince you and at least some members of this list of the 
> value of our study. Thats fine, and I can live with that. 
> Just drop me a note, if you do not want to be notified of the 
> results of our followup paper, and I will take you off my list.
> 
> However, some claims have been made here regarding the 
> process I used to carry out and evaluate this study and about 
> my scientific integrity, which I believe to be false. I will 
> at least try to address them in this mail, along with a few 
> more perspectives from my side.
> 
> On Wednesday 22 February 2006 13:09, Joe Armstrong (AL/EAB) wrote:
> [snip]
> > And what did I say here? -
> >
> > The complete post was:
> >
> >               If you use Erlang, why not tell
> >
> >                    http://www.plm.eecs.uni-kassel.de/parasurvey/
> >
> >               About it
> >
> >               /Joe
> >
> > Just compare this for a moment to what was posted to the LAM/MPI 
> > General User's Mailing List
> >
> > http://www.lam-mpi.org/MailArchives/lam/2005/10/11389.php
> >
> > I quote:
> > 	> More than 50 people have filled out the survey this far, and
> >
> > therefore I will
> >
> >       > be evaluating the results shortly (it will close in 
> two weeks,
> >
> > on November
> >
> >       > the 5th!). But before I do, please consider filling out the
> >
> > survey to make
> >
> >       > the results even more valuable. Of course, I will make the
> >
> > results available
> >
> >       > to everyone who participated. And before I forget 
> to mention it:
> >
> > two gift
> >
> >       > certificates from amazon.com are being awarded to 
> everyone who
> >
> > participated.
> >
> >       > Thank you for your cooperation,
> >       > Best Regards,
> >       > Michael Suess
> >
> > The bit about the gift certificates goes totally 
> unmentioned in your 
> > paper.
> 
> p.2, Survey Methodology:
> "If participants submitted their answers along with a working 
> e-mail address, they could win one of two 50$ gift 
> certificates from amazon.com."
> 
> 
> > You say  "An influential member of the Erlang community
> > requested members of their mailing list to show there 
> support of Erlang
> > by participating
> > in the Survey" - which is a false claim - I never said 
> anything about
> > support - I asked the
> > people on this list to tell you about their experiences.
> 
> This seems to be the single most important point, that most 
> people here are 
> complaining about, right? Reading through your mail again, I 
> admit that the 
> phrase I used in the paper is quite strong and I will make 
> sure to change it 
> in my followup paper. I will even go as far as to say: 
> 
> "The phrase is misleading, my mistake, sorry about it."
> 
> I would be very interested in how you or all the other people 
> on this list who 
> have been bothered by it would phrase it, though, as I cannot 
> leave out the 
> fact that you posted on this list entirely. Without this, 
> figure 4 gives a 
> very wrong impression, as the Erlang people were the only 
> ones (of the 
> systems mentioned in this figure) notified of the survey 
> (whether by you, or 
> by me does not make a difference here, as others have claimed). 
> 
> > And then you omit to say that gift certificates were 
> offered to members
> > of other mailing
> > lists, and finally when you get an unexpectly positive 
> response from the
> > members of the Erlang
> > mailing list you dismiss this this result since I asked 
> members of this
> > list to
> > participate in your survey.
> >
> > Usually in an academic paper it is consider de rigour to 
> describe your
> > experimental procedure.
> > Omitting to mention that you offered gift certificates to people who
> > filled in the
> > survey is rather strange since it is probably that it will bias the
> > results.
> >
> > Note that doing so you may well have biased the results in 
> favour of MPI
> > - MPI got the highest
> > rating among parallel programming systems - was this 
> because you offered
> > them gift certificates?
> >
> > If you are going to make unsubstantiated claims in your 
> paper about the
> > supposed
> > influence of any mailing that I made to this list - then 
> you should also
> > mention the
> > ways in which you other results might be biased.
> >
> > And by the way - you haven't sent me a gift certificate - 
> was the offer
> > only open
> > to people on the MPI list? - did the other people on this 
> list get any
> > gift certificates?
> 
> This paragraph made me very angry at first, then very sad. I 
> do not know how 
> long it took you to write it, but it took me about 10 seconds 
> to open the 
> report, press "find" and search for "gift". There is only one 
> result, and 
> this is the quote given above. And it is exactly where it is 
> supposed to be, 
> right under "Survey Methodology". Is it really too much to 
> ask of you to do 
> the same and actually invest these 10 seconds, before you 
> write 7 paragraphs 
> of false claims about my scientific methodology and try to 
> attack it on this 
> level?

 You're right - and I missed this part of the paper - sorry. 

 When I wrote this I had just seen the posting to the net which said:
 
 "two gift certificates from amazon.com are being awarded to everyone 
  who participated."

The paper said

   ".. they could win one of two 50$ gift certificates"
 
These mean different things. You said that my posting might have biased the results - this annoyed me and a lot of people on the Erlang list
who responded to your survey, and I annoy you by pointing out that
you have not accurately described your methods. 

It is not my intention to anger you, and I am sure that you did not
intend to annoy various members of the Erlang list - but it was my intention to point out the difference between facts and speculations
in your paper. It will be a better paper if you stick to the numbers, 
and leave off any attempt at guessing as to why they are as they are. 

I suggest you revise the paper a bit.

You could say:

<<
	- I sent the following post to [list them]  newsgroups

	" quote what you sent "


As a result of this a number of message asking people to participate in the survey were cross posted to a number of different groups.

It is impossible to estimate how the results of these messages biased
the responses from the different groups.

>> 



My personal opinions is that:

	- Offers of gift certificate has no effect on people filling
	  in the survey, and
	- My posting to the Erlang list has no effect (other than 
	  making people aware that there was a survey - I think the
	  response from the Erlang list would have been the same 
	  no matter who had sent the mail)  

[snipped]

/Joe

[can we put this bit of the argument to rest now and wave a few olive branches around - I have a program to write  ]