[erlang-questions] [ANN] Silly benchmarking

[Jeremy Ong]

I think comparison benchmarks are still useful. Even against a system
with no other load, knowing that X does the equivalent thing that Y
does but faster could provide initial direction in the implementation
of something new at least.

Agreed that benchmarks can be harmful if they aren't viewed
objectively though. We just need a caveat that the user measures his
or her own application as well!

On Tue, Apr 30, 2013 at 10:06 AM, Loïc Hoguin <essen@REDACTED> wrote:
> I'm sure *you* know these results might not be true for systems under load,
> but others might not. Not having both could cause more harm than good, and
> would not be interesting in that sense.
>
>
> On 04/30/2013 07:03 PM, Garrett Smith wrote:
>>
>> Not very interesting to you. Of course you can write whatever you
>> like. For *me* I was curious about some relative performance
>> characteristics. No religion here.
>>
>> On Tue, Apr 30, 2013 at 11:59 AM, Loïc Hoguin <essen@REDACTED> wrote:
>>>
>>> It's not very interesting unless the numbers are also available for
>>> systems
>>> under load.
>>>
>>>
>>> On 04/30/2013 06:57 PM, Jeremy Ong wrote:
>>>>
>>>>
>>>> I'd be very interested if we got a wiki going on one of these projects
>>>> with community updated numbers of various benchmark runs.
>>>>
>>>> For example:
>>>>
>>>> keylists vs orddicts vs dicts
>>>> fold vs recursion
>>>> fibonacci
>>>> all the other various standard benchmarks
>>>>
>>>> On Tue, Apr 30, 2013 at 9:42 AM, Garrett Smith <g@REDACTED> wrote:
>>>>>
>>>>>
>>>>> On Tue, Apr 30, 2013 at 9:35 AM, Michael Truog <mjtruog@REDACTED>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 04/30/2013 06:44 AM, Garrett Smith wrote:
>>>>>>>
>>>>>>>
>>>>>>> This is not an announcement of anything -- but [ANN] seems to flag
>>>>>>> "something I can maybe use" which does apply in this case :)
>>>>>>>
>>>>>>> Occasionally I wonder, "what's faster"? It's not often, but it
>>>>>>> happens.
>>>>>>>
>>>>>>> I've found the best way to answer this is to measure things.
>>>>>>>
>>>>>>> So I have this silly project:
>>>>>>>
>>>>>>> https://github.com/gar1t/erlang-bench
>>>>>>>
>>>>>>> It's not rigorous but it's simple and I can experiment quickly with
>>>>>>> different implementations. My goal is just to get a sense of things
>>>>>>> --
>>>>>>> not to formally prove anything.
>>>>>>>
>>>>>>> It's so trivial it's almost not worth sharing/reusing -- *however* it
>>>>>>> may provide value as a distributed repository for what people are
>>>>>>> interested in. As it's in github there's no ownership -- please feel
>>>>>>> free to fork and use for your own concerns!
>>>>>>>
>>>>>> You might want to look at erlbench here
>>>>>> https://github.com/okeuday/erlbench since it has the same basic
>>>>>> purpose, and
>>>>>> allows you to use different compilation methods now (through the
>>>>>> makefile
>>>>>> specifying an optimization level).  The erlbench project is also
>>>>>> ad-hoc, but
>>>>>> it has been enough to produce results in the past.
>>>>>>
>>>>>> The other option is trying to use basho_bench here
>>>>>> https://github.com/basho/basho_bench, if you are testing key/value
>>>>>> storage.
>>>>>
>>>>>
>>>>>
>>>>> Yes, but you'll notice how *easy* it is to use erlang-bench, which is
>>>>> nothing more than escript files with a 10 line include file.
>>>>>
>>>>> I'm an extraordinarily lazy person :)
>>>>>
>>>>> Though seriously, thanks for the references. If I was more concerned
>>>>> about benchmark integrity, those might be good options -- but this is
>>>>> just a sniff test approach to satisfy my curiosity about various
>>>>> topics.
>>>>>
>>>>> Garrett
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> erlang-questions@REDACTED
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questions@REDACTED
>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>>
>>>
>>>
>>> --
>>> Loďc Hoguin
>>> Erlang Cowboy
>>> Nine Nines
>>> http://ninenines.eu
>
>
>
> --
> Loïc Hoguin
>
> Erlang Cowboy
> Nine Nines
> http://ninenines.eu