<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Rick!<br>
<br>
On 08/14/2013 04:48 PM, Rick Reed wrote:<br>
</div>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div dir="ltr">
<div>Hi Patrik!</div>
<div><br>
</div>
And you want the requests in the same async queue to enforce
ordering per
<div>file descriptor or some other reason? It seems like
ordering isn't an issue</div>
<div>because the ultimately the file calls in erlang are
synchronous, and an app</div>
<div>would have to enforce ordering itself anyway (we do it by
sending all the</div>
<div>i/o for a file through a single proc and/or setting our own
per-file locks).</div>
<div><br>
</div>
</div>
</blockquote>
Yes, one example is process exit, where close definitely should not
be intermingled <br>
with other file operations from other threads that are ongoing. That
definitely happens if<br>
you round robin the file descriptors. I remember that there has been
other situations where <br>
the synchronous Erlang interface is not enough, but I can not for my
life remember them right now. <br>
Anyway, process exit is definitely one example :) <br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>For the app I'm debugging now, it turns out no scheme that
ties the port to</div>
<div>a particular thread is going to work. The system is
running at the limits of</div>
<div>the hardware, and the ports are long-lived. Only perfect
distribution of i/o</div>
<div>requests over the available threads prevents certain
threads from being</div>
<div>overloaded and backing up the i/o on the ports that map to
it.</div>
</div>
</blockquote>
Well, given the current design, I'm afraid a really good hash is the
best I can come up with :(<br>
<br>
The I/O should be rethought and rewritten once we have dirty
schedulers instead<br>
of the async threads!<br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I've been running a few of the systems overnight with a
patch that disables</div>
<div>keying in efile_drv. Now I'm getting a nice flat
distribution of i/o across the</div>
<div>async threads. Unfortunately, it hasn't completely solved
my problem, but</div>
<div>those systems are doing much better.</div>
</div>
</blockquote>
Yes, probably. It is not safe though, especially compressed files in
combination with <br>
processes getting exit (kill) signals during the file operations may
core the VM.<br>
<br>
With better distribution of the FD's maybe you can get as good
results as with <br>
the round robin without risks? <br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I'm just wondering if there's some other reason that I'm
missing (cache/mem</div>
<div>affinity, platform differences, etc.) for having to map
file descriptors to</div>
<div>particular threads.</div>
</div>
</blockquote>
I don't think it helps caches that much, it's far more threads than
cores anyway, so it's bound <br>
to generate inter-core communication regardless. <br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thanks for looking into this!</div>
</div>
</blockquote>
Thanks for reporting!<br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Rr</div>
</div>
</blockquote>
<br>
Cheers,<br>
Patrik<br>
<blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Wed, Aug 14, 2013 at 1:36 AM, Patrik
Nyblom <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:pan@erlang.org" target="_blank">pan@erlang.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Rick!
<div class="im"><br>
<br>
On 08/14/2013 02:21 AM, Rick Reed wrote:<br>
</div>
</div>
<div class="im">
<blockquote type="cite">
<div dir="ltr">I assume the reason for keying the file
requests is to prevent a single port from
<div>soaking up all the async threads?</div>
</div>
</blockquote>
</div>
Yes, and it's also important that requests for the same
file "descriptor" end up in she same async queue. So we
need to store a fixed key in the file descriptor
structure.<br>
<br>
I think I will hash the pointer to create the key, not
just shift away the "zero-bits", you never know which icky
patterns an allocator can create that will distribute the
jobs unevenly. The key will only be calculated upon
opening, so there will be minimal performance hit due to
the more complicated calculations.<br>
<br>
Thanks for reporting - this could cause severe performance
issues in applications!<br>
<br>
Cheers,<br>
Patrik
<div>
<div class="h5"><br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Rr</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote"> On Tue, Aug 13, 2013 at
4:52 AM, Lukas Larsson <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:lukas@erlang.org"
target="_blank">lukas@erlang.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> And
there it is, conclusive proof that I should
not be debugging Rickard's code before
lunch. <br>
<br>
Found the issue, will create a fix for it.
As a workaround for R16B you can use a prime
number as the number of async threads :)<span><font
color="#888888"><br>
<br>
Lukas</font></span>
<div>
<div><br>
<br>
<div>On 13/08/13 10:05, Lukas Larsson
wrote:<br>
</div>
<blockquote type="cite"> Sigh,
apparently I spoke too soon. <br>
<br>
I remembered incorrectly about the
change. It was in R16B that ErlDrvPort
became a ptr and it was an id before
R16B. Anyways, it is odd that the ptr
is 8 bit aligned on you system. On
mine (Ubuntu 13.04, x86_64) the ptrs
are not aligned and the load is nicely
distributed among async threads. If I
remember correctly you are using
FreeBSD on x86_64? I'll check if I can
reproduce the behavior you are seeing
on our FreeBSD machine. <br>
<br>
Lukas<br>
<br>
<div>On 13/08/13 09:40, Lukas Larsson
wrote:<br>
</div>
<blockquote type="cite"> Hello Rick!<br>
<br>
Which version of Erlang are you
using? From R16B (I think), the
ErlDrvPort datatype no longer is a
pointer to the port struct. Instead
it is the slot id into the port
table and those ids should contain
all values. I did a quick test on my
computer running the latest on maint
on github and seem to get a full
spread over all async threads. <br>
<br>
Lukas<br>
<br>
<div>On 13/08/13 05:40, Rick Reed
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">It looks to me as
though there's a bit of a
problem in the way efile_drv.c
generates the
<div>key that's used to select
an async driver queue. It
uses the address of the port
which</div>
<div>on our system is 8-byte
aligned. Meanwhile,
erl_async.c does a simple mod
operation</div>
<div>with the number of async
threads, so the number of
threads that can actually be
used</div>
<div>by file operations is 1/8th
of the number configured. I
suspect this isn't intended.</div>
<div><br>
</div>
<div>Rr</div>
<div><br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
erlang-bugs mailing list<br>
<a moz-do-not-send="true"
href="mailto:erlang-bugs@erlang.org">erlang-bugs@erlang.org</a><br>
<a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-bugs"
target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>