[erlang-questions] benchmarks game harsh criticism
Isaac Gouy
igouy2@REDACTED
Mon Dec 3 00:09:52 CET 2007
--- Ulf Wiger <ulf@REDACTED> wrote:
-snip-
> Not that I particularly want to weigh in on the discussion about the
> benchmarks - I think they are interesting, and since they are even
> advertised as "flawed", criticism should rather be directed towards
> those who read too much into them, (: but...
>
> Regarding the startup time of the Erlang runtime, even the longest
> running benchmarks are ridiculously short compared to the running
> time of the applications for which Erlang was designed. It is by no
> means unusual for an Erlang-based system to run for years before
> it is restarted, and from that perspective, a few seconds is indeed
> insignificant. Not that we don't consider startup/restart times - we
> do, since it contributes to the overall downtime that needs to be
> kept under ca 5-6 minutes/year. But that would be system restart
> time, since the systems are redundant, and that happens very,
> very seldomly.
>
> One of the things Erlang does to make it easier to keep the system
> running for years on end, is that each module can be reloaded at
> run-time, without having to restart any processes. During startup,
> the runtime system loads the modules needed, by searching the
> code loader path for the right file. For the core modules, loaded
> from the boot script, the path is set to point directly to the right
> modules (possibly preceded by directories identified from the
> command line), so there is not much searching going on, but
> once the user-level applications start, code loading time will
> depend on which modules are called for by the program.
>
> I have no idea how much this affects the startup time of the
> benchmarks. Those programs are small and usually don't
> call on that many modules, so my guess would be that it isn't
> such an important factor. I'm even leaning towards thinking that
> it's not such a bad thing for Erlang that the shootout includes
> startup time, since it is a likely weakness of Erlang, given its
> focus on non-stop applications. (:
At present, the only program with imports is partial-sums
-import(math,[pow/2,sin/1,cos/1]).
-import(lists,[foreach/2,zipwith/3]).
"Different design intentions - domain"
http://shootout.alioth.debian.org/gp4/miscfile.php?file=benchmarking&title=Flawed%20Benchmarks#domain
I think your better argument is:
Lesson learned: its nearly impossible to predict system performance
based on low-level micro benchmarks.
Failure to manage complexity often kills performance.
____________________________________________________________________________________
Be a better sports nut! Let your teams follow you
with Yahoo Mobile. Try it now. http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ
More information about the erlang-questions
mailing list