<div dir="ltr"><div>Hi Patrik!</div><div><br></div>And you want the requests in the same async queue to enforce ordering per<div>file descriptor or some other reason? It seems like ordering isn't an issue</div><div>because the ultimately the file calls in erlang are synchronous, and an app</div>
<div>would have to enforce ordering itself anyway (we do it by sending all the</div><div>i/o for a file through a single proc and/or setting our own per-file locks).</div><div><br></div><div>For the app I'm debugging now, it turns out no scheme that ties the port to</div>
<div>a particular thread is going to work. The system is running at the limits of</div><div>the hardware, and the ports are long-lived. Only perfect distribution of i/o</div><div>requests over the available threads prevents certain threads from being</div>
<div>overloaded and backing up the i/o on the ports that map to it.</div><div><br></div><div>I've been running a few of the systems overnight with a patch that disables</div><div>keying in efile_drv. Now I'm getting a nice flat distribution of i/o across the</div>
<div>async threads. Unfortunately, it hasn't completely solved my problem, but</div><div>those systems are doing much better.</div><div><br></div><div>I'm just wondering if there's some other reason that I'm missing (cache/mem</div>
<div>affinity, platform differences, etc.) for having to map file descriptors to</div><div>particular threads.</div><div><br></div><div>Thanks for looking into this!</div><div><br></div><div>Rr</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Aug 14, 2013 at 1:36 AM, Patrik Nyblom <span dir="ltr"><<a href="mailto:pan@erlang.org" target="_blank">pan@erlang.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Rick!<div class="im"><br>
<br>
On 08/14/2013 02:21 AM, Rick Reed wrote:<br>
</div></div><div class="im">
<blockquote type="cite">
<div dir="ltr">I assume the reason for keying the file requests is
to prevent a single port from
<div>soaking up all the async threads?</div>
</div>
</blockquote></div>
Yes, and it's also important that requests for the same file
"descriptor" end up in she same async queue. So we need to store a
fixed key in the file descriptor structure.<br>
<br>
I think I will hash the pointer to create the key, not just shift
away the "zero-bits", you never know which icky patterns an
allocator can create that will distribute the jobs unevenly. The key
will only be calculated upon opening, so there will be minimal
performance hit due to the more complicated calculations.<br>
<br>
Thanks for reporting - this could cause severe performance issues in
applications!<br>
<br>
Cheers,<br>
Patrik<div><div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Rr</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Tue, Aug 13, 2013 at 4:52 AM, Lukas Larsson <span dir="ltr"><<a href="mailto:lukas@erlang.org" target="_blank">lukas@erlang.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> And there it is,
conclusive proof that I should not be debugging Rickard's
code before lunch. <br>
<br>
Found the issue, will create a fix for it. As a workaround
for R16B you can use a prime number as the number of async
threads :)<span><font color="#888888"><br>
<br>
Lukas</font></span>
<div>
<div><br>
<br>
<div>On 13/08/13 10:05, Lukas Larsson wrote:<br>
</div>
<blockquote type="cite"> Sigh, apparently I spoke too
soon. <br>
<br>
I remembered incorrectly about the change. It was in
R16B that ErlDrvPort became a ptr and it was an id
before R16B. Anyways, it is odd that the ptr is 8
bit aligned on you system. On mine (Ubuntu 13.04,
x86_64) the ptrs are not aligned and the load is
nicely distributed among async threads. If I
remember correctly you are using FreeBSD on x86_64?
I'll check if I can reproduce the behavior you are
seeing on our FreeBSD machine. <br>
<br>
Lukas<br>
<br>
<div>On 13/08/13 09:40, Lukas Larsson wrote:<br>
</div>
<blockquote type="cite"> Hello Rick!<br>
<br>
Which version of Erlang are you using? From R16B
(I think), the ErlDrvPort datatype no longer is a
pointer to the port struct. Instead it is the slot
id into the port table and those ids should
contain all values. I did a quick test on my
computer running the latest on maint on github and
seem to get a full spread over all async threads.
<br>
<br>
Lukas<br>
<br>
<div>On 13/08/13 05:40, Rick Reed wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">It looks to me as though there's
a bit of a problem in the way efile_drv.c
generates the
<div>key that's used to select an async driver
queue. It uses the address of the port
which</div>
<div>on our system is 8-byte aligned.
Meanwhile, erl_async.c does a simple mod
operation</div>
<div>with the number of async threads, so the
number of threads that can actually be used</div>
<div>by file operations is 1/8th of the number
configured. I suspect this isn't intended.</div>
<div><br>
</div>
<div>Rr</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
erlang-bugs mailing list<br>
<a href="mailto:erlang-bugs@erlang.org">erlang-bugs@erlang.org</a><br>
<a href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>
<br></blockquote></div><br></div>