[erlang-questions] benchmarks game harsh criticism

[Isaac Gouy]

--- David Hopwood <david.hopwood@REDACTED> wrote:

> It doesn't depend (as a first-order approximation) on how fast the
> machine is, because you're running the benchmarks for fixed problem
> sizes. A problem size that runs for 0.10 s on a 2.2 Ghz AMD Sempron,
> for example, is too small to give you reliable results on any
> machine,
> regardless of its speed.
> 
> Most of the unreliability I'm talking about is not due to timing
> measurement error -- it's due to dependency of the results on factors
> that don't tell you anything useful about the general performance of
> the language implementation, such as how the code from a particular
> compilation happened to be placed in cache.

Again you haven't provided any example of an anomaly in the benchmarks
game measurements (let alone Erlang program measurements) that might
correspond to this "unreliability".

Maybe I just have less time on my hands, this is too much like chasing
phantoms for me. By all means pile-up criticisms in the benchmarks game
discussion forum.

I'll leave you with the last word, and with the hope that you will say
whether or not you think the proportion of startup/shutdown in the
Erlang program measurements I've reported supports your claim that "the
resulting bias is quite large".


      ____________________________________________________________________________________
Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ