[erlang-questions] Cost of doing +sbwt?

[Paul Davis]

On Fri, Sep 4, 2015 at 3:09 AM, Jesper Louis Andersen
<jesper.louis.andersen@REDACTED> wrote:
>
> On Fri, Sep 4, 2015 at 12:39 AM, Paul Davis <paul.joseph.davis@REDACTED>
> wrote:
>>
>> Sooo....
>>
>> Distributed systems are hard.
>
>
> One have to celebrate the small victories. You know what is causing the
> trouble in the first place, and this is an important milestone. Surely,
> auditd will impose an overhead in the running system, but this impact it is
> having seems excessively high for an audit system to me.
>

Yeah, its interesting to see that it seems to affect 17.5 so much more
than R14B01 but beyond that I don't plan on digging into the kernel to
satisfy any more curiosity on that front. What's most amusing is that
we spent so much time testing 17.5 and then auditd got shoved out
between testing and deployment. For some value of amusing I guess.

> The slightly higher CPU is to be expected, given that 15+ and onwards does
> spinning waiting for runq entries. One nice trick is to enable monitor
> events (erlang:system_monitor, see Basho's riak_sysmon as well). These will
> warn you about long schedules. In particular long schedules on system calls
> that are expected to return quickly :) Perhaps this will help in the future
> when someone changes assumptions underneath the setup.
>
>
> --
> J.

I agree. The slightly higher CPU of a couple percent or so is well
within reason. Also good call on the long schedule monitors. I'm gonna
add a ticket now for that so I don't forget to add it now that 17.5 is
a thing for us.

Thanks for all your help and suggestions. I knew it was definitely a
stretch with "17.5 makes our clusters sad" and thankfully we had a
heads up engineer catch the auditd difference. Distributed systems are
hard.

Cheers,
Paul Davis