[erlang-questions] Joe Armstrong's proposal for a test/benchmark implementation

[Shaun Kruger]

On Fri, 2011-06-17 at 08:40 -0400, Frédéric Trottier-Hébert wrote:
> To test real 'real world' solutions, you need to give full freedom to the benchmarkee to build their own environment and an app the way they want it. You might be testing a LAMP stack compared to a LYME (Linux Yaws MySQL Erlang) stack, and there will be plenty of suggestions as to how to make one faster than the other.
> 
> The rabbit hole only gets deeper at each step. I do support the effort though. I think it would be better than microbenchmarks, although the tests would likely still not be reliable to judge on the performance of an application X.
> 

The one test that every benchmark misses is the "How well does this
language help me ship software" test.  Selecting for performance is
great when you're trying to avoid selecting a language that is going to
bog you down.  However, once you determine that a language is going to
work "well enough" you should select based on what you're trying to do
and accept that developer time is more expensive than hardware.

Develop a benchmark that takes into account the cost of the time spent
by development and admin teams to accomplish the benchmark and you will
be the smartest man alive. :)

That being said, I started my HTTP proxy project 9 months ago.  Erlang
has enabled me to deliver it faster and with more features than any
other language I could have selected.  

When someone suggests that I shouldn't use language X because it doesn't
have good benchmarks I instantly find myself wondering how long it has
been since that person did anything useful.

Shaun
https://github.com/skruger/Surrogate