[erlang-questions] Who is interested in an Erlang Performance Improvement Book?

[Loïc Hoguin]

On 08/14/2015 02:12 PM, Alexander Shorin wrote:
> On Fri, Aug 14, 2015 at 2:58 PM, Loïc Hoguin <essen@REDACTED> wrote:
>>
>> In other words: we have fantastic test tools to ensure we don't add new bugs
>> with each commit; what would you do to make sure you don't kill performance
>> with each commit?
>>
>> A good answer to that alone would be worth a thousand books on performance.
>
> Golang had used special benchmark test suite which ran on each commit
> against various configurations[1] and it was quite trivial to detect
> which commit caused performance and for what case, but suddenly it
> doesn't works anymore. Some bits web archive still remember[2] about
> how it looked like.
>
> But in general, solution of this problem is the same: make benchmarks
> as yet another test suite and track how long it being executed, not
> just ok/error status. At some point turning integration tests into
> benchmarks might be good idea.

The time spent to run X is the worst kind of benchmark value you can 
rely on though.

In Erlang for example you may replace some code with a NIF (or even a 
BIF), effectively making these benchmarks faster, only to find out later 
that it makes the whole system slower when you have 100k processes using 
it concurrently.

Or perhaps a piece of code will run faster, but with double memory 
usage. Or the latency suffers. Etc.

One thing for sure is that you need a benchmark test suite, but how to 
set it up to make sure the system as a whole benefits from all the 
changes is a guide I've never seen written.

-- 
Loïc Hoguin
http://ninenines.eu
Author of The Erlanger Playbook,
A book about software development using Erlang