[erlang-questions] benchmarks game harsh criticism

[Ulf Wiger]

2007/11/29, Brent Fulgham <bfulg@REDACTED>:
>
> On Nov 28, 2007, at 7:42 PM, David Hopwood wrote:
> The times that take longer than a few seconds don't affect my point
> that there is systematic bias against language implementations with
> significant startup/shutdown times.
>
> You say this as though significant startup/shutdown times should be
> considered acceptable.  I disagree -- take for example SBCL or SML/NJ,
> both of which have sizeable runtimes and yet manage to produce very
> good times.

Not that I particularly want to weigh in on the discussion about the
benchmarks - I think they are interesting, and since they are even
advertised as "flawed", criticism should rather be directed towards
those who read too much into them, (:  but...

Regarding the startup time of the Erlang runtime, even the longest
running benchmarks are ridiculously short compared to the running
time of the applications for which Erlang was designed. It is by no
means unusual for an Erlang-based system to run for years before
it is restarted, and from that perspective, a few seconds is indeed
insignificant. Not that we don't consider startup/restart times - we
do, since it contributes to the overall downtime that needs to be
kept under ca 5-6 minutes/year. But that would be system restart
time, since the systems are redundant, and that happens very,
very seldomly.

One of the things Erlang does to make it easier to keep the system
running for years on end, is that each module can be reloaded at
run-time, without having to restart any processes. During startup,
the runtime system loads the modules needed, by searching the
code loader path for the right file. For the core  modules, loaded
from the boot script, the path is set to point directly to the right
modules (possibly preceded by directories identified from the
command line), so there is not much searching going on, but
once the user-level applications start, code loading time will
depend on which modules are called for by the program.

I have no idea how much this affects the startup time of the
benchmarks. Those programs are small and usually don't
call on that many modules, so my guess would be that it isn't
such an important factor. I'm even leaning towards thinking that
it's not such a bad thing for Erlang that the shootout includes
startup time, since it is a likely weakness of Erlang, given its
focus on non-stop applications. (:

BR,
Ulf W