[erlang-questions] Joe Armstrong's proposal for a test/benchmark implementation

[Frédéric Trottier-Hébert]

That's an issue with the idea of 'real world'. Real world will vary a lot depending on who you're talking to. Oh yes, my sessions can be stored in memory on memcached, no issue with that, just buy more memory. 

Hm, you're testing this on this kind of hardware? too bad, this kind of software really runs better on a different kind of it! 

Did they configure this right? because this is pretty important.

Oh sorry, my implementation works better on some other OS...

To test real 'real world' solutions, you need to give full freedom to the benchmarkee to build their own environment and an app the way they want it. You might be testing a LAMP stack compared to a LYME (Linux Yaws MySQL Erlang) stack, and there will be plenty of suggestions as to how to make one faster than the other.

The rabbit hole only gets deeper at each step. I do support the effort though. I think it would be better than microbenchmarks, although the tests would likely still not be reliable to judge on the performance of an application X.

--
Fred Hébert
http://www.erlang-solutions.com



On 2011-06-17, at 08:14 AM, Tim Watson wrote:

>> 
>> I remember myself saying that we could take the dscussion to the list and and arrive at a basic idea how to do this.
>> 
> 
> If you want it to be "real world" then your application should include
> talking to a database or other external service (pref over a network).
> I like the idea of having a kind of log-in-then-do-something approach.
> 
> What are you going to use for benchmarking? Tsung? Something else?
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions