[erlang-questions] benchmarks game harsh criticism
Fri Nov 30 19:49:53 CET 2007
--- David Hopwood <david.hopwood@REDACTED> wrote:
> It doesn't depend (as a first-order approximation) on how fast the
> machine is, because you're running the benchmarks for fixed problem
> sizes. A problem size that runs for 0.10 s on a 2.2 Ghz AMD Sempron,
> for example, is too small to give you reliable results on any
> regardless of its speed.
> Most of the unreliability I'm talking about is not due to timing
> measurement error -- it's due to dependency of the results on factors
> that don't tell you anything useful about the general performance of
> the language implementation, such as how the code from a particular
> compilation happened to be placed in cache.
Again you haven't provided any example of an anomaly in the benchmarks
game measurements (let alone Erlang program measurements) that might
correspond to this "unreliability".
Maybe I just have less time on my hands, this is too much like chasing
phantoms for me. By all means pile-up criticisms in the benchmarks game
I'll leave you with the last word, and with the hope that you will say
whether or not you think the proportion of startup/shutdown in the
Erlang program measurements I've reported supports your claim that "the
resulting bias is quite large".
Be a better sports nut! Let your teams follow you
with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ
More information about the erlang-questions