[erlang-questions] benchmarks game harsh criticism (was Learning Erlang from the scratch)

[Isaac Gouy]

On Nov 23, 2007 1:26 PM, Bengt Kleberg wrote:

> this is seriously off topic for erlang-questions, so i would
recommend
> each and every one of you to stop reading now.

Was it off topic when you chose to air your criticism on
erlang-questions back in September, or is it only the reply to your
criticism that is off topic? :-)



> kvw(*) is not a benchmark report. it is a paper about experiments 
> on how to do benchmarks. 

Maybe you misread ?

- "This paper describes some basic /experiments to see how fast/
various popular scripting and user-interface languages run on a
spectrum of representative tasks."

- "This paper describes /experiments to compare the performance/ of
scripting languages ... on a set of representative computational
tasks." 



-snip-
> to see this it is necessary to vary the input to such an extent as
> to find the dramatic runtime changes. the shootout does not do this.

There's no intention to increase input values until each program
exhausts resources and fails - that much is true. Despite that, we do
include programs that exhaust stack or heap, and fail, at larger input
values.



> isaac gouy has previously stated that the shootout is not, and shall
> not be, about the kind of wide spectrum of inputs that kvw recommends
> investigating. now he is instead saying that the shootout is better 
> than what kvw recommends for this kind of investigations.

You suggest that "kvw recommends" something about a wide spectrum of
inputs _ I've searched in vain for that recommendation - please quote
their exact words.

Kernighan and Van Wyk did write "Varying the problem size helped us to
detect unusual runtime effects ..." and we can see that they varied the
problem size by < ~10x

imo You are distorting Kernighan and Van Wyk to lend authority to your
own viewpoint. 


      ____________________________________________________________________________________
Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ