[erlang-questions] benchmarks game harsh criticism

[David Hopwood]

Brent Fulgham wrote:
> On Nov 28, 2007, at 7:42 PM, David Hopwood wrote:
> 
>> The times that take longer than a few seconds don't affect my point
>> that there is systematic bias against language implementations with
>> significant startup/shutdown times.
> 
> You say this as though significant startup/shutdown times should be
> considered acceptable.

I think they should. This obviously depends on the intended domain of
application of the language (and on whether it supports a way of
submitting tasks to be run by an existing VM process).

Startup/shutdown time is an aspect of language implementation performance
and it's perfectly reasonable to measure it, but this cost should not be
attributed to all of the other benchmarks, because that's not realistic
for typical uses of those language implementations.

> I disagree -- take for example SBCL or SML/NJ, both of which have
> sizeable runtimes and yet manage to produce very good times.

That's beside the point. It can be more difficult to optimize startup
time for one language implementation than another, or the implementation
developers may have considered this to be a low priority relative to
other considerations.

-- 
David Hopwood