What does scheduler/aux mean in msacc?
Fernando Benavides
elbrujohalcon@REDACTED
Tue Jul 7 13:52:50 CEST 2020
Thanks Jesper,
I understand all that but I already run the tests in more than 32 different
AWS machines… in groups of 2 OTP22 / 2 OTP21 machines each time… and the
results are impressively consistent. Every single time I get the results I
posted before (different %s of course, but not significantly different and
it's always the same difference ratio between the different OTP versions).
I'm really really convinced that this is not a fluke due to virtualization.
It's something that our system is doing that got worse in OTP22, for sure…
but I don't know where to look.
Cheers!
On Tue, Jul 7, 2020 at 12:40 PM Jesper Louis Andersen <
jesper.louis.andersen@REDACTED> wrote:
> You should run on the same machines, and preferably at the same time (back
> to back). When you are running under a hypervisor, you are cooperating with
> other users of the hardware, and they can wreak serious havoc on your
> benchmarks. I've seen 20% fluctuations in performance on AWS hardware due
> to this. The usual problems stems from how the CPUs handle caches and if
> there are other users who move a lot of data around, you will see a lower
> performance due to this.
>
> My advice will be to reconstruct the problem on local hardware. If you
> cannot do this, then chances are you are looking at some AWS artifact
> rather than a change in OTP behavior.
>
>
>
> On Mon, Jul 6, 2020 at 5:14 PM Fernando Benavides <elbrujohalcon@REDACTED>
> wrote:
>
>> Yeah… the curious thing is that both machines are the same kind of AWS
>> virtual machines… And I had 2 running OTP22 and 2 running OTP21… and both
>> OTP22 machines showed xen_hypercall_event_channel_op while both OTP21
>> didn't (actually… they did, but with a very smaller % of overhead). And,
>> BTW, the same happened with smp_call_function_many (more overhead
>> associated with that function in OTP22 than OTP21).
>>
>> I'll keep digging, but if anybody has any idea of what to look for, I
>> would appreciate it a lot.
>>
>> Thanks
>>
>> On Mon, Jul 6, 2020 at 5:05 PM Max Lapshin <max.lapshin@REDACTED> wrote:
>>
>>> First post from the internet about xen_hypercall_event_channel_op:
>>> https://www.scylladb.com/2018/06/21/impact-virtualization-database/
>>>
>>> TL;DR: metal server better than virtual
>>>
>>> On Mon, Jul 6, 2020 at 5:26 PM Fernando Benavides <
>>> elbrujohalcon@REDACTED> wrote:
>>>
>>>> I'll add another data point…
>>>> As Lukas suggested, I used *perf* on the running systems and this is
>>>> what I've got…
>>>>
>>>> For OTP22:
>>>> 2.33% 8_scheduler beam.smp [.] process_main
>>>>
>>>> * …the rest of the schedulers running process_main…* 1.57%
>>>> 6_scheduler beam.smp [.] process_main
>>>> 0.34% 6_scheduler [kernel.kallsyms] [k]
>>>> xen_hypercall_event_channel_op
>>>> 0.32% 7_scheduler [kernel.kallsyms] [k]
>>>> xen_hypercall_event_channel_op
>>>> 0.25% 14_scheduler beam.smp [.] copy_struct_x
>>>> 0.25% 2_scheduler beam.smp [.] copy_struct_x
>>>> 0.25% 9_scheduler [kernel.kallsyms] [k]
>>>> xen_hypercall_event_channel_op
>>>> 0.23% 11_scheduler [kernel.kallsyms] [k]
>>>> xen_hypercall_event_channel_op
>>>> 0.22% 11_scheduler beam.smp [.] copy_shallow_x
>>>> 0.22% 9_scheduler libc-2.17.so [.]
>>>> __memcpy_ssse3_back
>>>> 0.20% 13_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.20% 3_scheduler [kernel.kallsyms] [k]
>>>> xen_hypercall_event_channel_op
>>>> 0.19% 10_scheduler beam.smp [.] erts_cmp_compound
>>>>
>>>> For OTP21:
>>>> 2.45% 4_scheduler beam.smp [.] process_main
>>>>
>>>> * …the rest of the schedulers running process_main…* 1.91%
>>>> 5_scheduler beam.smp [.] process_main
>>>> 0.45% 15_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.27% 10_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.27% 3_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.27% 7_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.25% 1_scheduler beam.smp [.] copy_struct_x
>>>> 0.25% 2_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.23% 2_scheduler beam.smp [.] copy_struct_x
>>>> 0.23% 7_scheduler beam.smp [.] copy_struct_x
>>>> 0.20% 4_scheduler beam.smp [.] copy_struct_x
>>>> 0.20% 6_scheduler [kernel.kallsyms] [k] __lock_text_start
>>>> 0.18% 14_scheduler beam.smp [.] copy_struct_x
>>>>
>>>> I'm not sure what any of those functions are, or why
>>>> xen_hypercall_event_channel_op is so much popular in OTP22, tho… ��
>>>>
>>>> On Mon, Jul 6, 2020 at 3:42 PM Max Lapshin <max.lapshin@REDACTED>
>>>> wrote:
>>>>
>>>>> In a nutshell there is a non-reproducible fluctuation of measurement:
>>>>> one subsystem takes 10% instead 8%, another 21% instead 23%.
>>>>>
>>>>> Totally there seems to be no change.
>>>>>
>>>>> It is extremely hard to make reliable measurement that helps to find
>>>>> 8% to 10% change.
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> <https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
>>>> Brujo Benavides
>>>> about.me/elbrujohalcon
>>>> <https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
>>>>
>>>
>>
>> --
>>
>> <https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
>> Brujo Benavides
>> about.me/elbrujohalcon
>> <https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
>>
>
>
> --
> J.
>
--
<https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
Brujo Benavides
about.me/elbrujohalcon
<https://about.me/elbrujohalcon?promo=email_sig&utm_source=product&utm_medium=email_sig&utm_campaign=gmail_api&utm_content=thumb>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200707/9aa87bc6/attachment.htm>
More information about the erlang-questions
mailing list