[erlang-questions] Joe Armstrong's proposal for a test/benchmark implementation

[Tim Watson]

>
> To test real 'real world' solutions, you need to give full freedom to the benchmarkee to build their own environment and an app the way they want it. You might be testing a LAMP stack compared to a LYME (Linux Yaws MySQL Erlang) stack, and there will be plenty of suggestions as to how to make one faster than the other.
>
> The rabbit hole only gets deeper at each step. I do support the effort though. I think it would be better than microbenchmarks, although the tests would likely still not be reliable to judge on the performance of an application X.
>

Perhaps what is needed is a framework that makes it easier to
construct a larger/broader kind of benchmark, that accounts for things
like the OS & hw environment, configuration, and does so both for the
(test) clients and the application(s) being tested. A lot of effort
has gone into automating deployments and infrastructure configuration
in recent years - capistrano/fabric, puppet and chef, etc - and
various efforts (such as Vagrant) into using virtualisation to make
testing against multiple architectures/environments easier. In fact
IIRC erlang-solutions have built something that does CI/Build of
Erlang/OTP across various configurations and environments
automatically, am I right?

So I wonder if bringing together some of these things - build +
deployment automation, environment/infrastructure configuration,
testing + benchmarking tools such as Tsung and basho_bench, etc - into
a "something" that make it possible (or even "not-utterly-painful"
perhaps) to simplify and/or automate these things for the purposes of
testing stuff, might be very useful.

I'm pretty sure that the "something" which will make this easier is
fairly composite, but it would be nice if you didn't have start from
scratch when trying to do something like this (comparing different
tools, platforms, architectures and so on). It might also be
impossibly complicated to do right. :)