[erlang-questions] performance drop in R15A?

[Nico Kruber]

On Monday 16 April 2012 22:11:20 Yiannis Tsiouris wrote:
> On Mon, Apr 16, 2012 at 04:03:49PM +0200, Nico Kruber wrote:
> > Thank you for the tips, indeed, by using the +sbt option I was able to
> > improve our benchmark results to a certain extend - not to the full
> > R14B04 value though :(
> > I achieved the best results together with the +swt low and very_low
> > option, so I would probably go with "low". I also experienced with +rg
> > and +scl but these didn't influence the results too much.
> > 
> > I also attached the cpuinfo from the test system (basically an 8-core
> > Opteron 2376 with 8GB RAM).
> > 
> > using Scalaris "make bench":
> >            wall clock throughput // latency
> > 
> > R14B04:                         999  //  40
> > 
> >  "+swt low"                    1097  //  36
> >  "+swt very_low"               1101  //  36
> > 
> > R15B: (same as "+sbt u")        679  //  59
> > 
> >  "+scl true"                    682  //  59
> >  "+sbt db"                      836  //  48
> >  "+sbt ns"                      828  //  48
> >  "+sbt ts"                      840  //  48
> >  "+sbt ps"                      816  //  49
> >  "+sbt s"                       822  //  49
> >  "+sbt nnts"                    825  //  48
> >  "+sbt nnps"                    834  //  48
> >  "+sbt tnnps"                   830  //  48
> >  "+scl true +sbt db"            827  //  48
> >  "+scl true +sbt db +swt low"  1029  //  39
> >  "+sbt db +swt low"            1019  //  39
> >  "+sbt db +swt very_low"       1038  //  39
> >  "+swt low"                     764  //  52
> >  "+sbt db +swt low +rg 4"      1008  //  40
> >  "+sbt db +swt low +rg 8"      1042  //  38
> >  "+sbt db +swt low +rg 16"     1011  //  40
> > 
> > Recall that this creates 4 erlang VMs with the same (given) start-up
> > options.
> Hi,
> 
> First of all, thanks for sharing all these experiences with us! It's been
> really interesting. I have one question though: how stable are your
> results? I notice that some of the timings are really close, i.e. the
> variance is around 10-20%. Do you think that these measurements are
> reliable-enough to make valid assumptions?

Regarding stability of the benchmarks in the plots I attached a while ago: 
these fluctuate because the machine is not only used for benchmarking. The 
trend is clear though.

Regarding the values mentioned above, they are relatively stable, e.g. see 
"+sbt db" and "+sbt tnnps" which (according to the man page) are the same. 
There is less than 1% difference.
On the other hand, the difference between the options is dramatic (and 
reproducable)!
Here are the results of 10 runs of "+sbt db +swt low":

R14B04:
1096 // 36.5
1088 // 36.7
1095 // 36.5
1094 // 36.5
1102 // 36.3
1100 // 36.4
1101 // 36.3
1095 // 36.5
1097 // 36.5
1089 // 36.7

R15B:
1026 // 39.0
1026 // 39.0
1026 // 39.0
1019 // 39.2
1012 // 39.5
1016 // 39.4
1028 // 38.9
1025 // 39.0
1021 // 39.2
1019 // 39.3

-> as you can see, with these two options on, the bench results between R14B04 
and R15B do not differ much any more (avg: R14B04 1096, R15B 1022) but R15B is 
around 7% slower. The different results on each Erlang version however do not 
differ that much (R14B04 stddev 4.7 - R15B stddev 5.2; both less than 1%)

Nico