[erlang-questions] Joe Armstrong's proposal for a test/benchmark implementation

[Loïc Hoguin]

On 06/17/2011 04:34 PM, Dmitrii Dimandt wrote:
> 
> On Jun 17, 2011, at 3:30 PM, Cyryl Płotnicki-Chudyk wrote:
> 
>> 1up
>>
>> As competitors I would see Ruby on Rails , ASP.NET and some php stuff
>> as these are among the most popular ones.
>> The app might involve validating login via the webvervice/rest/json
>> stuff and later, after the login, getting some data to display and to
>> be edited from the postgres/mysql db
>>
>> The deployment cost should be measured somehow imho, e.g. how much
>> time a admin needs to deploy new machine with each of the solutions.
>> And I mean the whole time, installing all necessary frameworks/gems
>> etc.
> 
> If we take Fred's words into account, then for each system we test we might need to make installation as easy and painless as possible so that ideally anyone could just download the code, deploy it and have a benchmark on their hands relevant to their hardware

While not a bad idea per se, this is unfortunately not really
representative as only small applications require no or few system
changes to make them perform well. If you benchmark scaling to 1k
simultaneous connections then it's all fine and well, but Erlang can
scale to millions of simultaneous connections on the same server and
that's not something easily tested.

General benchmarks like you propose are more useful as a testing method
than actual performance comparison. For example you could detect
performance issues on one or another platform that you don't have at
your disposal. They're also a very good marketing tool as most people
trust them without question. That's why you get many meaningless
benchmarks in blogs for example.

So sure, that's not necessarily a bad idea, but if you are going to tell
people to install your app + benchmark to get an idea of the performance
of different products, you should also sensibilize them about this and
give a few pointers on how they might improve the performance of all
products on their system through system tweaks and more.

This is pretty much a full-time job.

Regardless I'm still looking forward to see php and ruby getting their
ass kicked even though the benchmarks would probably be biased.

-- 
Loïc Hoguin
Dev:Extend