[erlang-questions] benchmarks game harsh criticism

[Isaac Gouy]

--- David Hopwood <david.hopwood@REDACTED> wrote:

-snip-
> Then let me be more specific.

Thank you for being more specific.


> From the FAQ at <http://shootout.alioth.debian.org/gp4/faq.php>:
> 
> # CPU Time means program usr+sys time (in seconds) which includes the
> # time taken to startup and shutdown the program. For language
> # implementations that use a Virtual Machine the CPU Time includes
> # the time taken to startup and shutdown the VM.
> 
> This is an elementary error, sufficiently serious that it's not
> enough just for the FAQ to mention it in passing. It systematically
> biases the results against language implementations with a
significant
> startup/shutdown time, or other fixed overheads. Combined with the
> fact that most of the benchmarks only run for a few seconds, the
> resulting bias is quite large.

Specifically how large is the resulting bias?

Is it large enough that we should reassess the 97.6 seconds that the
HiPE program takes for fannkuch down to the 5.99 seconds taken by the C
program, or only large enough that we should reassess it to 97.0
seconds?

I do remember a comment from Ulf Wiger to the effect that the Erlang VM
seemed to reach peak performance after 24 hours of constant load, which
he thought very appropriate given the intended uses.


-snip-
> The other main factor that makes the shootout almost useless for
> language comparison, is the widely differing amount of optimization
> effort put into the code submissions.

Firstly, I think that may be a criticism of benchmarks in general, I
don't recall seeing published benchmarks with a statement of how much
optimization effort was put into each program. 

Secondly, I don't think you know that there was a widely differing
optimization effort - it's just an assumption.




      ____________________________________________________________________________________
Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ