[erlang-questions] benchmarks game harsh criticism
Thu Nov 29 09:22:09 CET 2007
On Nov 28, 2007, at 10:31 PM, Isaac Gouy wrote:
>> Another basic mistake is that there is no indication of the variation
>> in timing between benchmark runs. At least, not without digging a bit
>> deeper: the excluded "Haskell GHC #4" result for N=9 on nsieve is
>> 1.12 s, but in the full results for nsieve-bits, the result for N=9
>> exactly the same program run by the same language implementation is
>> 0.80 s.
> Are you sure the basic mistake is not yours?
> Gentoo : Intel® Pentium® 4
> nsieve,ghc,4,9, 1.124,
> nsievebits,ghc,4,9, 1.112
Oh --- he must have been comparing the runs on the two test machines
(the Gentoo Intel Pentium 4
and the Debian AMD Sempron):
Debian: AMD Sempron:
So I guess you could argue that there is a 40% variation if you
compare the runs
from two different hardware platforms.
More information about the erlang-questions