[erlang-questions] benchmarks game harsh criticism

[Isaac Gouy]

--- Bengt Kleberg <bengt.kleberg@REDACTED> wrote:

> greetings,
> 
> so, you think that the 3 quotes from "Timing Trials" does not
> recommend 
> a certain method to show additional insights, instead they describe a
> 
> certain method to show additional insights. that is ok with me. i can
> 
> change from recomend to describe, without losing track of the target.
> 
> moreover, i will assume that you mention "C runtimes appears
> absolutely 
> horizontal" [for this test], as one example when "Timing Trials"
> write 
> "anomalous behavior that deserves further attention." that is no
> problem 
> with me. (if you mean that this is the one and only thing ever to 
> deserve more investigation, i would like an explanation on how you
> have 
> managed to arrive at that idea.)
> 
> 
> anyway then, we seem to agree on the following: "Timing Trials"
> observes 
> that it is a good idea to have sufficiently many measuring points
> during 
> benchmarking to be able to spot anomalous behaviour. i think the 
> shootout does not do this and that it would be a good thing if it
> did. 
> you do not want the shootout do this, for reasons never explained.


Do you agree that the benchmarks game has a range of input values ~10x
to ~100x, in comparison to mostly < 10x in "Timing Trials"?


Do you agree that the benchmarks game range of input values /has/ been
used to spot anomalous behaviour - binary-trees memory usage not
increasing for some programs?


      ____________________________________________________________________________________
Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ