[erlang-questions] Joe Armstrong's proposal for a test/benchmark implementation

[Dmitrii Dimandt]

>> 
>> To test real 'real world' solutions, you need to give full freedom to the benchmarkee to build their own environment and an app the way they want it. You might be testing a LAMP stack compared to a LYME (Linux Yaws MySQL Erlang) stack, and there will be plenty of suggestions as to how to make one faster than the other.
>> 
>> The rabbit hole only gets deeper at each step. I do support the effort though. I think it would be better than microbenchmarks, although the tests would likely still not be reliable to judge on the performance of an application X.
>> 
> 
> Perhaps what is needed is a framework that makes it easier to
> construct a larger/broader kind of benchmark, that accounts for things
> like the OS & hw environment, configuration, and does so both for the
> (test) clients and the application(s) being tested. A lot of effort
> has gone into automating deployments and infrastructure configuration
> in recent years - capistrano/fabric, puppet and chef, etc - and
> various efforts (such as Vagrant) into using virtualisation to make
> testing against multiple architectures/environments easier. In fact
> IIRC erlang-solutions have built something that does CI/Build of
> Erlang/OTP across various configurations and environments
> automatically, am I right?
> 
> So I wonder if bringing together some of these things - build +
> deployment automation, environment/infrastructure configuration,
> testing + benchmarking tools such as Tsung and basho_bench, etc - into
> a "something" that make it possible (or even "not-utterly-painful"
> perhaps) to simplify and/or automate these things for the purposes of
> testing stuff, might be very useful.
> 
> I'm pretty sure that the "something" which will make this easier is
> fairly composite, but it would be nice if you didn't have start from
> scratch when trying to do something like this (comparing different
> tools, platforms, architectures and so on). It might also be
> impossibly complicated to do right. :)


Hmm... I never thought of it this way :) It sounds like a lot of work, but it would definitely be nice to arrive to something like that in the end.