[erlang-bugs] efile_drv & async thread key

[Rick Reed]

Hi Patrik!

And you want the requests in the same async queue to enforce ordering per
file descriptor or some other reason?  It seems like ordering isn't an issue
because the ultimately the file calls in erlang are synchronous, and an app
would have to enforce ordering itself anyway (we do it by sending all the
i/o for a file through a single proc and/or setting our own per-file locks).

For the app I'm debugging now, it turns out no scheme that ties the port to
a particular thread is going to work.  The system is running at the limits
of
the hardware, and the ports are long-lived.  Only perfect distribution of
i/o
requests over the available threads prevents certain threads from being
overloaded and backing up the i/o on the ports that map to it.

I've been running a few of the systems overnight with a patch that disables
keying in efile_drv.  Now I'm getting a nice flat distribution of i/o
across the
async threads.  Unfortunately, it hasn't completely solved my problem, but
those systems are doing much better.

I'm just wondering if there's some other reason that I'm missing (cache/mem
affinity, platform differences, etc.) for having to map file descriptors to
particular threads.

Thanks for looking into this!

Rr


On Wed, Aug 14, 2013 at 1:36 AM, Patrik Nyblom <pan@REDACTED> wrote:

>  Hi Rick!
>
>
> On 08/14/2013 02:21 AM, Rick Reed wrote:
>
> I assume the reason for keying the file requests is to prevent a single
> port from
> soaking up all the async threads?
>
> Yes, and it's also important that requests for the same file "descriptor"
> end up in she same async queue. So we need to store a fixed key in the file
> descriptor structure.
>
> I think I will hash the pointer to create the key, not just shift away the
> "zero-bits", you never know which icky patterns an allocator can create
> that will distribute the jobs unevenly. The key will only be calculated
> upon opening, so there will be minimal performance hit due to the more
> complicated calculations.
>
> Thanks for reporting - this could cause severe performance issues in
> applications!
>
> Cheers,
> Patrik
>
>
>  Rr
>
>
>  On Tue, Aug 13, 2013 at 4:52 AM, Lukas Larsson <lukas@REDACTED> wrote:
>
>>  And there it is, conclusive proof that I should not be debugging
>> Rickard's code before lunch.
>>
>> Found the issue, will create a fix for it. As a workaround for R16B you
>> can use a prime number as the number of async threads :)
>>
>> Lukas
>>
>>
>> On 13/08/13 10:05, Lukas Larsson wrote:
>>
>> Sigh, apparently I spoke too soon.
>>
>> I remembered incorrectly about the change. It was in R16B that ErlDrvPort
>> became a ptr and it was an id before R16B. Anyways, it is odd that the ptr
>> is 8 bit aligned on you system. On mine (Ubuntu 13.04, x86_64) the ptrs are
>> not aligned and the load is nicely distributed among async threads. If I
>> remember correctly you are using FreeBSD on x86_64? I'll check if I can
>> reproduce the behavior you are seeing on our FreeBSD machine.
>>
>> Lukas
>>
>> On 13/08/13 09:40, Lukas Larsson wrote:
>>
>> Hello Rick!
>>
>> Which version of Erlang are you using? From R16B (I think), the
>> ErlDrvPort datatype no longer is a pointer to the port struct. Instead it
>> is the slot id into the port table and those ids should contain all values.
>> I did a quick test on my computer running the latest on maint on github and
>> seem to get a full spread over all async threads.
>>
>> Lukas
>>
>> On 13/08/13 05:40, Rick Reed wrote:
>>
>> It looks to me as though there's a bit of a problem in the way
>> efile_drv.c generates the
>> key that's used to select an async driver queue.  It uses the address of
>> the port which
>> on our system is 8-byte aligned.  Meanwhile, erl_async.c does a simple
>> mod operation
>> with the number of async threads, so the number of threads that can
>> actually be used
>> by file operations is 1/8th of the number configured.  I suspect this
>> isn't intended.
>>
>>  Rr
>>
>>
>>
>> _______________________________________________
>> erlang-bugs mailing listerlang-bugs@REDACTED://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>>
>> _______________________________________________
>> erlang-bugs mailing listerlang-bugs@REDACTED://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>>
>> _______________________________________________
>> erlang-bugs mailing listerlang-bugs@REDACTED://erlang.org/mailman/listinfo/erlang-bugs
>>
>>
>>
>
>
> _______________________________________________
> erlang-bugs mailing listerlang-bugs@REDACTED://erlang.org/mailman/listinfo/erlang-bugs
>
>
>
> _______________________________________________
> erlang-bugs mailing list
> erlang-bugs@REDACTED
> http://erlang.org/mailman/listinfo/erlang-bugs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-bugs/attachments/20130814/cfeae62f/attachment.htm>