[erlang-questions] benchmarks game harsh criticism

[Brent Fulgham]

On Nov 28, 2007, at 10:31 PM, Isaac Gouy wrote:
>> Another basic mistake is that there is no indication of the variation
>> in timing between benchmark runs. At least, not without digging a bit
>> deeper: the excluded "Haskell GHC #4" result for N=9 on nsieve is
>> 1.12 s, but in the full results for nsieve-bits, the result for N=9
> on
>> exactly the same program run by the same language implementation is
>> 0.80 s.
>
> Are you sure the basic mistake is not yours?
>
> Gentoo : Intel® Pentium® 4
>
> nsieve,ghc,4,9, 1.124,
>
> nsievebits,ghc,4,9, 1.112

Oh --- he must have been comparing the runs on the two test machines  
(the Gentoo Intel Pentium 4
and the Debian AMD Sempron):

Debian: AMD Sempron:

nsieve,ghc,4,9, 0.740
nsievebits,ghc,4,9,0.744

So I guess you could argue that there is a 40% variation if you  
compare the runs
from two different hardware platforms.

-Brent