[erlang-questions] benchmarks game harsh criticism

[Isaac Gouy]

--- Ulf Wiger <ulf@REDACTED> wrote:
-snip- 
> Not that I particularly want to weigh in on the discussion about the
> benchmarks - I think they are interesting, and since they are even
> advertised as "flawed", criticism should rather be directed towards
> those who read too much into them, (:  but...
> 
> Regarding the startup time of the Erlang runtime, even the longest
> running benchmarks are ridiculously short compared to the running
> time of the applications for which Erlang was designed. It is by no
> means unusual for an Erlang-based system to run for years before
> it is restarted, and from that perspective, a few seconds is indeed
> insignificant. Not that we don't consider startup/restart times - we
> do, since it contributes to the overall downtime that needs to be
> kept under ca 5-6 minutes/year. But that would be system restart
> time, since the systems are redundant, and that happens very,
> very seldomly.
> 
> One of the things Erlang does to make it easier to keep the system
> running for years on end, is that each module can be reloaded at
> run-time, without having to restart any processes. During startup,
> the runtime system loads the modules needed, by searching the
> code loader path for the right file. For the core  modules, loaded
> from the boot script, the path is set to point directly to the right
> modules (possibly preceded by directories identified from the
> command line), so there is not much searching going on, but
> once the user-level applications start, code loading time will
> depend on which modules are called for by the program.
> 
> I have no idea how much this affects the startup time of the
> benchmarks. Those programs are small and usually don't
> call on that many modules, so my guess would be that it isn't
> such an important factor. I'm even leaning towards thinking that
> it's not such a bad thing for Erlang that the shootout includes
> startup time, since it is a likely weakness of Erlang, given its
> focus on non-stop applications. (:

At present, the only program with imports is partial-sums

-import(math,[pow/2,sin/1,cos/1]).
-import(lists,[foreach/2,zipwith/3]).


"Different design intentions - domain"
http://shootout.alioth.debian.org/gp4/miscfile.php?file=benchmarking&title=Flawed%20Benchmarks#domain


I think your better argument is: 

• Lesson learned: it’s nearly impossible to predict system performance
based on low-level ”micro” benchmarks.
• Failure to manage complexity often kills performance.



      ____________________________________________________________________________________
Be a better sports nut!  Let your teams follow you 
with Yahoo Mobile. Try it now.  http://mobile.yahoo.com/sports;_ylt=At9_qDKvtAbMuh1G1SQtBI7ntAcJ