[erlang-questions] [ANN] Silly benchmarking

[Michael Truog]

I like the idea of having a site like http://benchmarksgame.alioth.debian.org/ that is more specific to Erlang.  Ideally we could test fun stuff, like comparing actors in Scala/Akka and JActor (https://github.com/laforge49/JActor) with Erlang.  However, to keep stuff transparent and useful we would need to document hardware and avoid short runs that skew results.  To have a serious site requires a monetary commitment and I am not sure who would be willing to pursue that (Ericsson might be the best organization to pursue this and show how Erlang shines).  Either way, having the source code available is a requirement for making sure the tests are repeatable, so I think we can pursue that direction in the meantime.

One other benchmarking project I am aware of is here:
http://www.softlab.ntua.gr/release/bencherl/index.html

So there is probably a lot of individual testing efforts that could be combined, to provide definitive data for decision making.  That probably requires discussion and coordination at a higher-level.

On 04/30/2013 10:02 AM, Garrett Smith wrote:
> This is sort of my thinking with the github project:
>
> - Better than a wiki because you can run the files
> - Easily forked and modified -- so people can share and modify code trivially
> - Distributed is better than centralized!
>
> There's probably a temptation to view anything like this as
> authoritative -- it of course is not and cannot be (e.g. the three
> stupid files I have are totally arbitrary). I think distributed source
> like this is a good push-back for that temptation.
>
> On Tue, Apr 30, 2013 at 11:57 AM, Jeremy Ong <jeremy@REDACTED> wrote:
>> I'd be very interested if we got a wiki going on one of these projects
>> with community updated numbers of various benchmark runs.
>>
>> For example:
>>
>> keylists vs orddicts vs dicts
>> fold vs recursion
>> fibonacci
>> all the other various standard benchmarks
>>
>> On Tue, Apr 30, 2013 at 9:42 AM, Garrett Smith <g@REDACTED> wrote:
>>> On Tue, Apr 30, 2013 at 9:35 AM, Michael Truog <mjtruog@REDACTED> wrote:
>>>> On 04/30/2013 06:44 AM, Garrett Smith wrote:
>>>>> This is not an announcement of anything -- but [ANN] seems to flag
>>>>> "something I can maybe use" which does apply in this case :)
>>>>>
>>>>> Occasionally I wonder, "what's faster"? It's not often, but it happens.
>>>>>
>>>>> I've found the best way to answer this is to measure things.
>>>>>
>>>>> So I have this silly project:
>>>>>
>>>>> https://github.com/gar1t/erlang-bench
>>>>>
>>>>> It's not rigorous but it's simple and I can experiment quickly with
>>>>> different implementations. My goal is just to get a sense of things --
>>>>> not to formally prove anything.
>>>>>
>>>>> It's so trivial it's almost not worth sharing/reusing -- *however* it
>>>>> may provide value as a distributed repository for what people are
>>>>> interested in. As it's in github there's no ownership -- please feel
>>>>> free to fork and use for your own concerns!
>>>>>
>>>> You might want to look at erlbench here https://github.com/okeuday/erlbench since it has the same basic purpose, and allows you to use different compilation methods now (through the makefile specifying an optimization level).  The erlbench project is also ad-hoc, but it has been enough to produce results in the past.
>>>>
>>>> The other option is trying to use basho_bench here https://github.com/basho/basho_bench, if you are testing key/value storage.
>>> Yes, but you'll notice how *easy* it is to use erlang-bench, which is
>>> nothing more than escript files with a 10 line include file.
>>>
>>> I'm an extraordinarily lazy person :)
>>>
>>> Though seriously, thanks for the references. If I was more concerned
>>> about benchmark integrity, those might be good options -- but this is
>>> just a sniff test approach to satisfy my curiosity about various
>>> topics.
>>>
>>> Garrett
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
> .
>