[erlang-questions] [ANN] Silly benchmarking

[Garrett Smith]

I'm curious what the OTP team does (in the lab) to vet various
implementation options.

On Tue, Apr 30, 2013 at 12:35 PM, Fred Hebert <mononcqc@REDACTED> wrote:
> A rather simple trick to simulate load is to have a few busy-looping
> processes on the VM, and give it a higher priority than the rest. It
> should eat time more than anything else and make it look like there's a
> lot of contention for schedulers, with reduced execution time for what
> you want to measure.
>
> I've used it a few times, although I am not 100% sure it's super
> representative o a real system being under load.
>
> On 04/30, Garrett Smith wrote:
>> You have a point. Though "load" would be pretty arbitrary -- I could
>> see some load-xxx.escript files that did terrible things while the
>> other scripts were run. But it would depend on what else is running on
>> your system. And your system architecture. And your system hardware.
>> Waiiit a minute, this just got real complicated!
>>
>> So, don't make decisions unless you have reasonably complete
>> information, whatever that means :)
>>
>> On Tue, Apr 30, 2013 at 12:06 PM, Loïc Hoguin <essen@REDACTED> wrote:
>> > I'm sure *you* know these results might not be true for systems under load,
>> > but others might not. Not having both could cause more harm than good, and
>> > would not be interesting in that sense.
>> >
>> >
>> > On 04/30/2013 07:03 PM, Garrett Smith wrote:
>> >>
>> >> Not very interesting to you. Of course you can write whatever you
>> >> like. For *me* I was curious about some relative performance
>> >> characteristics. No religion here.
>> >>
>> >> On Tue, Apr 30, 2013 at 11:59 AM, Loïc Hoguin <essen@REDACTED> wrote:
>> >>>
>> >>> It's not very interesting unless the numbers are also available for
>> >>> systems
>> >>> under load.
>> >>>
>> >>>
>> >>> On 04/30/2013 06:57 PM, Jeremy Ong wrote:
>> >>>>
>> >>>>
>> >>>> I'd be very interested if we got a wiki going on one of these projects
>> >>>> with community updated numbers of various benchmark runs.
>> >>>>
>> >>>> For example:
>> >>>>
>> >>>> keylists vs orddicts vs dicts
>> >>>> fold vs recursion
>> >>>> fibonacci
>> >>>> all the other various standard benchmarks
>> >>>>
>> >>>> On Tue, Apr 30, 2013 at 9:42 AM, Garrett Smith <g@REDACTED> wrote:
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Apr 30, 2013 at 9:35 AM, Michael Truog <mjtruog@REDACTED>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>> On 04/30/2013 06:44 AM, Garrett Smith wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> This is not an announcement of anything -- but [ANN] seems to flag
>> >>>>>>> "something I can maybe use" which does apply in this case :)
>> >>>>>>>
>> >>>>>>> Occasionally I wonder, "what's faster"? It's not often, but it
>> >>>>>>> happens.
>> >>>>>>>
>> >>>>>>> I've found the best way to answer this is to measure things.
>> >>>>>>>
>> >>>>>>> So I have this silly project:
>> >>>>>>>
>> >>>>>>> https://github.com/gar1t/erlang-bench
>> >>>>>>>
>> >>>>>>> It's not rigorous but it's simple and I can experiment quickly with
>> >>>>>>> different implementations. My goal is just to get a sense of things
>> >>>>>>> --
>> >>>>>>> not to formally prove anything.
>> >>>>>>>
>> >>>>>>> It's so trivial it's almost not worth sharing/reusing -- *however* it
>> >>>>>>> may provide value as a distributed repository for what people are
>> >>>>>>> interested in. As it's in github there's no ownership -- please feel
>> >>>>>>> free to fork and use for your own concerns!
>> >>>>>>>
>> >>>>>> You might want to look at erlbench here
>> >>>>>> https://github.com/okeuday/erlbench since it has the same basic
>> >>>>>> purpose, and
>> >>>>>> allows you to use different compilation methods now (through the
>> >>>>>> makefile
>> >>>>>> specifying an optimization level).  The erlbench project is also
>> >>>>>> ad-hoc, but
>> >>>>>> it has been enough to produce results in the past.
>> >>>>>>
>> >>>>>> The other option is trying to use basho_bench here
>> >>>>>> https://github.com/basho/basho_bench, if you are testing key/value
>> >>>>>> storage.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Yes, but you'll notice how *easy* it is to use erlang-bench, which is
>> >>>>> nothing more than escript files with a 10 line include file.
>> >>>>>
>> >>>>> I'm an extraordinarily lazy person :)
>> >>>>>
>> >>>>> Though seriously, thanks for the references. If I was more concerned
>> >>>>> about benchmark integrity, those might be good options -- but this is
>> >>>>> just a sniff test approach to satisfy my curiosity about various
>> >>>>> topics.
>> >>>>>
>> >>>>> Garrett
>> >>>>> _______________________________________________
>> >>>>> erlang-questions mailing list
>> >>>>> erlang-questions@REDACTED
>> >>>>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> erlang-questions mailing list
>> >>>> erlang-questions@REDACTED
>> >>>> http://erlang.org/mailman/listinfo/erlang-questions
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Loďc Hoguin
>> >>> Erlang Cowboy
>> >>> Nine Nines
>> >>> http://ninenines.eu
>> >
>> >
>> >
>> > --
>> > Loïc Hoguin
>> >
>> > Erlang Cowboy
>> > Nine Nines
>> > http://ninenines.eu
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions