[erlang-questions] benchmarks game harsh criticism

[Bengt Kleberg]

greetings,

(personal message to brent: it is nice to see you in this discussion. 
you come across as ''the good cop'' from the shootout.)

on the shootout email list some years ago the problem of too short 
runtimes was discussed. one solution was to have a fixed the runtime,
and measure the input for this runtime. i even wrote and submitted a
program that would find the right input given a sought-after runtime and
a percentage (max deviation). since this was not what i wanted from the
shootout i never pushed the concept. still, it is an alternative.

on the other hand, limiting the shootout to the lowest common 
denominator, that is the opposite of what i want from shootout.
it is possible to allow a wide range of entries provided they are shown 
on a graph instead of as a number. that is what i want. it is not 
necessary to have every language getting a time for every input if they 
are graphed. you can remove the too fast entries and the too slow 
entries, the difference in performance is possible to see anyway.


bengt
Those were the days...
    EPO guidelines 1978: "If the contribution to the known art resides
    solely in a computer program then the subject matter is not
    patentable in whatever manner it may be presented in the claims."


On 11/29/07 09:14, Brent Fulgham wrote:
...deleted

> If you are concerned with the variation in the sub-second results, it 
> would be reasonable
> to cull out the worst implementations and perform tests for higher 
> values of N.  But that
> would prevent a wide range of entries, so to some extent it is a matter 
> of limiting
> ourselves to the lowest common denominator.
> 

...deleted

> -Brent
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions