[erlang-questions] Proper benchmarking

[Vance Shipley]

On Fri, Dec 14, 2007 at 10:01:00AM +0000, Joel Reymont wrote:
}  I'm gathering information for an extensive chapter on efficiency  
}  optimization. I will also try benchmarking various approaches in other  
}  chapters, e.g. straight binary work vs. pickler combinators, different  
}  types of interaction with a linked-in driver or gb_trees vs ets.
}  
}  Do you have any tips or suggestions apart from those given in the  
}  Efficiency Guide?

(Apologies in advance for the lengthy post)

Joel's benchmarking questions made me wonder about estone.  Back 
in the deep dark past there was a benchmark called "estone".  You 
could compare different emulator versions or platforms using it's
estone ratings.  

The FAQ currently has this to say:

	http://www.erlang.org/faq/faq.html#AEN795

The last reference to estone I could find is:

	http://www.erlang.org/pipermail/erlang-questions/2003-August/009557.html

... which says that estone is now part of the emulator test suite
used with the test_server application:

	http://www.erlang.org/doc/apps/test_server

So I had my first look at the test_server application.  Let me just
say that instant gratification was not the order of the day here.
There is a webtool front end (vts) which was a good starting point
however it absolutely insists on having a configuration file to
start with and there are none to be found in the distribution.
After a while I came to realize that the specification files (*.spec)
refered to in test_server are a superset of the configuration files
refered to in common_test.  Now there do happen to be several .spec
files in R12B-0:

	~/lib/edoc/test/edoc.spec
	~/lib/lib/inviso/test/inviso.spec
	~/lib/lib/lib/megaco/test/megaco.spec
	~/lib/lib/lib/lib/lib/percept/test/percept.spec
	~/lib/lib/lib/lib/lib/lib/snmp/test/snmp.spec

So where are the test specifications for OTP?  I found them here:

   http://www.erlang.org/download/test_server/emulator-test-2004-05-26.tar.gz
   http://www.erlang.org/download/test_server/stdlib-test-2004-05-26.tar.gz

These appear to be the missing test suites for the emulator and stdlib.
So I had a go at running the emulator test suites (after make, emake)
using the specification file emulator.spec for the configuration file 
which vts requires.  So far so good, it runs the tests.

However all the tests fail because the test suies do not export all/0.
It seems like there has been a change from all(suite) to all() in the 
callback API.  It is still documented as all(suite) with the exception 
of the example which follows how it seems to be implemented which is 
allow/0.  So I patched the emulator test suites and was able to run it.

	297 cases executed (of 297)
	215 successful
	 82 failed
	  0 skipped

One of the test suites was the estone I so fondly remember however it
was sadly one of those which failed.  It managed a number of tests but
failed on large_dataset_work/1.

The FAQ points us at Björn Gustavsso's home page for benchmarks
and from there I learn that the new benchmark is bstone (Björ's stone?)
which I find here:

	http://www.erlang.org/download/bstone.tar.z

This is nice and simple as it once was:

	$ erl           
	Erlang (BEAM) emulator version 5.6 [source] [smp:2] [async-threads:0] [kernel-poll:false]

	Eshell V5.6  (abort with ^G)
	1> c(bstone).
	{ok,bstone}
	2> bstone:run().
	lists(6400)
	msgp(1515)
	msgp_medium(1527)
	msgp_huge(52)
	pattern(1046)
	trav(2834)
	large_dataset_work(1193)
	large_local_dataset_work(1174)
	alloc(3710)
	bif_dispatch(1623)
	binary_h(581)
	ets(342)
	generic(7977)
	int_arith(4157)
	float_arith(5526)
	fcalls(882)
	timer(2312)
	links(30)
	
	[{seconds,3.98346},{stones,98543}]

Nice.  For the record though I did find it bombed out when started with
a thread pool:

	$ erl +A 32
	Erlang (BEAM) emulator version 5.6 [source] [smp:2] [async-threads:32] [kernel-poll:false]
	
	Eshell V5.6  (abort with ^G)
	1> bstone:run().
	lists(6400)
	msgp(1515)
	msgp_medium(1527)
	msgp_huge(52)
	pattern(1046)
	trav(2834)
	large_dataset_work(1193)
	large_local_dataset_work(1174)
	alloc(3710)
	bif_dispatch(1623)
	binary_h(581)
	ets(342)
	generic(7977)
	int_arith(4157)
	float_arith(5526)
	fcalls(882)
	timer(2312)
	links(30)
	
	=ERROR REPORT==== 14-Dec-2007::15:09:36 ===
	Error in process <0.12396.0> with exit value: {noproc,[{erlang,link,[<0.12327.0>]},{bstone,lproc,4},{bstone,lproc,1}]}
	

So the questions remain:

	Is there a bug(s) in test_server/common_test?

	Where are the up to date test suites for OTP?

	Is estone deprecated in favour of bstone?

	Is there a newer version of bstone?


	-Vance