[erlang-questions] benchmarks game harsh criticism
Fri Nov 30 21:17:53 CET 2007
--- Bengt Kleberg <> wrote:
> quotes from "Timing Trials, or, the Trials of Timing" as per request:
> "* Memory-related issues and the effects of memory hierarchies are
> pervasive: how memory is managed, from hardware caches to garbage
> collection, can change runtimes dramatically. Yet users have no
> control over most aspects of memory management. "
> "We started to construct a table with three dimensions: task,
> programming language, and machine. Eventually we added the size of
> problem solved by the program as a fourth dimension, and we changed
> presentation from tables to graphs. Varying the problem size helped
> to detect unusual runtime effects, while a graphical presentation
> highlights patterns and trends in runtime instead of individual
> performance scores."
> "We designed tests whose runtime should grow linearly with the size
> the problem: runtime = mÂ?size + b. Thus, if we choose size to be
> enough to justify ignoring the fixed overhead (b), the log-log plot
> should show a straight line of unit slope. Exceptions indicate
> behavior that deserves further attention."
(Incidentally what they meant by anomalous behaviour is something like
this: "the line connecting C runtimes appears absolutely horizontal.
... This happens because the optimizer eliminates the entire loop,
replacing it by sum = n.")
Let me suggest to you that the paragraphs you quote are descriptions
Here's a recommendation: "... we advise all who want to know which
version of a program will run faster to construct test programs and
find out the truth for their language processor and machine."
Here's another: "It does seem wise to take all such experiments
including these with a large grain of salt."
> Now it is your turn. could you quote the exact words where they say
> that 4 inputs and a spread of x10 is good enough?
There is no such statement, nor have I claimed that there is: I have
described the spread shown in the tests - "we can see that they varied
the problem size by < ~10x"
Be a better sports nut! Let your teams follow you
with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ
More information about the erlang-questions