[erlang-questions] performance drop in R15A?

[Kenneth Lundin]

On 12/5/11, Nico Kruber <kruber@REDACTED> wrote:
> Is there any news on this?
>
> The performance drop of the recent R15 checkout compared to R14B04 in our
> application is now at _30%_ compared to a drop of "only" 15% in R15A!
>
When working with R15B we are of course performing a number of
bencmarks to keep track of the performance.

Whe always strive to improve general performance in each release AND
OUR BENCHMARKS INDICATE THAT R15B IS FASTER IN GENERAL THAN R14B04.

It is however always possible that some specific operations or
combinations of operations get slower while others are optimized.

Regarding the findings for this particular application we think the
reason for slowdown is a very frequent but non typical usage pattern
that now becomes visible as a performance bottleneck.


/ Kenneth, Erlang/OTP Ericsson

>
> Nico
>
>
> On Friday 02 December 2011 11:26:56 Nico Kruber wrote:
>> Tonight, I ran the tests with an erlang checkout from around 16:00
>> yesterday
>> (erlang-otp-OTP_R15A-126-g0290bed.tar.gz).
>> This includes the fix Björn-Egil was referring to,
>> i.e. 'rickard/pix-mutex/OTP-9723'.
>>
>>
>> Following are the results or our "make bench" of Scalaris:
>>
>> ** transactions/s , i.e. "wall clock throughput  : xxx" **
>> R14B04: ~950, R15A (ge21ff9b): ~800, R15A (g0290bed): ~635
>>
>> ** latency, i.e. "wall clock avg. latency: xxx ms" **:
>> R14B04: ~42ms, R15A (ge21ff9b): ~50ms, R15A (g0290bed): ~60ms
>>
>>
>> Note: This "make bench" command sets up a Scalaris ring consisting of 4
>> nodes, each in a single VM. They all communicate through tcp.
>>
>>
>> Additionally, we also measure our performance through our Java-API which
>> accesses Erlang via JInterface using
>> "SCALARIS_BENCHMARKS="100 6 9 15 18" make java-bench"
>>
>> ** number of transactions per second for "Benchmark incrementing an
>> integer key (read+write):"
>> R14B04: ~310, R15A (ge21ff9b): ~200, R15A (g0290bed): ~140
>>
>>
>> I saw some new bug fixes for the scheduler in the recent commits, but with
>> the checkout from 6971b96 (just now), the results are still the same as
>> with g0290bed :(
>>
>>
>>
>> Regards,
>> Nico
>>
>> On Wednesday 30 November 2011 17:05:29 Nico Kruber wrote:
>> > thank you
>> > We'll try again tomorrow
>> >
>> > Nico
>> >
>> > On Wednesday 30 November 2011 16:30:05 Lukas Larsson wrote:
>> > > Hello!
>> > >
>> > > I was able to verify you findings on my machine for R14B04 and R15A.
>> > > The problem is however fixed (I even get a 15% throughput
>> > > improvement on your
>> > > bench) in our latest internal master which will be pushed to github
>> > > later
>> > > this evening. So if you could pull the latest from github tomorrow
>> > > and
>> > > verify my findings that would be great!
>> > >
>> > > Lukas
>> > >
>> > > On Wed, Nov 30, 2011 at 11:36 AM, Nico Kruber <kruber@REDACTED> wrote:
>> > > > Is it possible, that some debug option is on by default in the
>> > > > R15A
>> > > > tagged git
>> > > > checkout?
>> > > > We build the source using the following commands (on a x86_64
>> > > > system
>> > > > with 8 cores):
>> > > >
>> > > > ./otp_build autoconf
>> > > > ./configure --prefix=/home/buildbot/apps/erlang-R15A
>> > > > make
>> > > > make install
>> > > >
>> > > > and then compiled our application
>> > > > (http://code.google.com/p/scalaris/)
>> > > > using
>> > > > R15A.
>> > > >
>> > > > With our internal benchmarks ("make bench") showed a drop of
>> > > > around
>> > > > 15%
>> > > > in performance compared to R14B04. We were actually hoping for
>> > > > an
>> > > > increased performance reading about the memory allocation
>> > > > improvements...
>> > > >
>> > > > Is there some switch (during configuration or build) influencing
>> > > > the
>> > > > performance?
>> > > >
>> > > > output of R14B04 erl:
>> > > > ------------------------------------------------------------
>> > > > Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8]
>> > > > [async-
>> > > > threads:0] [hipe] [kernel-poll:false]
>> > > >
>> > > > Eshell V5.8.5  (abort with ^G)
>> > > > 1>
>> > > > ------------------------------------------------------------
>> > > >
>> > > > output of R15A erl:
>> > > > ------------------------------------------------------------
>> > > > Erlang R15A (erts-5.9.pre) [source] [64-bit] [smp:8:8] [rq:8]
>> > > > [async-
>> > > > threads:0] [hipe] [kernel-poll:false]
>> > > >
>> > > > Eshell V5.9.pre  (abort with ^G)
>> > > > 1>
>> > > > ------------------------------------------------------------
>> > > >
>> > > >
>> > > > Regards
>> > > > Nico Kruber