[erlang-questions] benchmarks game harsh criticism
Fri Nov 30 06:59:40 CET 2007
Isaac Gouy wrote:
> --- David Hopwood <david.hopwood@REDACTED> wrote:
>> BTW, anyone who has spent much time on benchmarking knows that you
>> cannot get reliable results from runs this short. I'm astonished that
>> I am having to argue this point.
> You state that as an absolute - afaict it would depend on how fast the
> machine was and the resolution of the time measurements.
It doesn't depend (as a first-order approximation) on how fast the
machine is, because you're running the benchmarks for fixed problem
sizes. A problem size that runs for 0.10 s on a 2.2 Ghz AMD Sempron,
for example, is too small to give you reliable results on any machine,
regardless of its speed.
Most of the unreliability I'm talking about is not due to timing
measurement error -- it's due to dependency of the results on factors
that don't tell you anything useful about the general performance of
the language implementation, such as how the code from a particular
compilation happened to be placed in cache.
More information about the erlang-questions