<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Hi Rick!<br>
      <br>
      On 08/14/2013 04:48 PM, Rick Reed wrote:<br>
    </div>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      <div dir="ltr">
        <div>Hi Patrik!</div>
        <div><br>
        </div>
        And you want the requests in the same async queue to enforce
        ordering per
        <div>file descriptor or some other reason?  It seems like
          ordering isn't an issue</div>
        <div>because the ultimately the file calls in erlang are
          synchronous, and an app</div>
        <div>would have to enforce ordering itself anyway (we do it by
          sending all the</div>
        <div>i/o for a file through a single proc and/or setting our own
          per-file locks).</div>
        <div><br>
        </div>
      </div>
    </blockquote>
    Yes, one example is process exit, where close definitely should not
    be intermingled <br>
    with other file operations from other threads that are ongoing. That
    definitely happens if<br>
    you round robin the file descriptors. I remember that there has been
    other situations where <br>
    the synchronous Erlang interface is not enough, but I can not for my
    life remember them right now. <br>
    Anyway, process exit is definitely one example :)  <br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>For the app I'm debugging now, it turns out no scheme that
          ties the port to</div>
        <div>a particular thread is going to work.  The system is
          running at the limits of</div>
        <div>the hardware, and the ports are long-lived.  Only perfect
          distribution of i/o</div>
        <div>requests over the available threads prevents certain
          threads from being</div>
        <div>overloaded and backing up the i/o on the ports that map to
          it.</div>
      </div>
    </blockquote>
    Well, given the current design, I'm afraid a really good hash is the
    best I can come up with :(<br>
    <br>
    The I/O should be rethought and rewritten once we have dirty
    schedulers instead<br>
    of the async threads!<br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>I've been running a few of the systems overnight with a
          patch that disables</div>
        <div>keying in efile_drv.  Now I'm getting a nice flat
          distribution of i/o across the</div>
        <div>async threads.  Unfortunately, it hasn't completely solved
          my problem, but</div>
        <div>those systems are doing much better.</div>
      </div>
    </blockquote>
    Yes, probably. It is not safe though, especially compressed files in
    combination with <br>
    processes getting exit (kill) signals during the file operations may
    core the VM.<br>
    <br>
    With better distribution of the FD's maybe you can get as good
    results as with <br>
    the round robin without risks? <br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>I'm just wondering if there's some other reason that I'm
          missing (cache/mem</div>
        <div>affinity, platform differences, etc.) for having to map
          file descriptors to</div>
        <div>particular threads.</div>
      </div>
    </blockquote>
    I don't think it helps caches that much, it's far more threads than
    cores anyway, so it's bound <br>
    to generate inter-core communication regardless. <br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>Thanks for looking into this!</div>
      </div>
    </blockquote>
    Thanks for reporting!<br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div><br>
        </div>
        <div>Rr</div>
      </div>
    </blockquote>
    <br>
    Cheers,<br>
    Patrik<br>
    <blockquote
cite="mid:CA+SuFX2hUtvOspa71SOf5BSVSnoAC6XL6+6MV+2yBXpdmGP=CA@mail.gmail.com"
      type="cite">
      <div class="gmail_extra">
        <br>
        <br>
        <div class="gmail_quote">On Wed, Aug 14, 2013 at 1:36 AM, Patrik
          Nyblom <span dir="ltr"><<a moz-do-not-send="true"
              href="mailto:pan@erlang.org" target="_blank">pan@erlang.org</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000">
              <div>Hi Rick!
                <div class="im"><br>
                  <br>
                  On 08/14/2013 02:21 AM, Rick Reed wrote:<br>
                </div>
              </div>
              <div class="im">
                <blockquote type="cite">
                  <div dir="ltr">I assume the reason for keying the file
                    requests is to prevent a single port from
                    <div>soaking up all the async threads?</div>
                  </div>
                </blockquote>
              </div>
              Yes, and it's also important that requests for the same
              file "descriptor" end up in she same async queue. So we
              need to store a fixed key in the file descriptor
              structure.<br>
              <br>
              I think I will hash the pointer to create the key, not
              just shift away the "zero-bits", you never know which icky
              patterns an allocator can create that will distribute the
              jobs unevenly. The key will only be calculated upon
              opening, so there will be minimal performance hit due to
              the more complicated calculations.<br>
              <br>
              Thanks for reporting - this could cause severe performance
              issues in applications!<br>
              <br>
              Cheers,<br>
              Patrik
              <div>
                <div class="h5"><br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div><br>
                      </div>
                      <div>Rr</div>
                    </div>
                    <div class="gmail_extra"><br>
                      <br>
                      <div class="gmail_quote"> On Tue, Aug 13, 2013 at
                        4:52 AM, Lukas Larsson <span dir="ltr"><<a
                            moz-do-not-send="true"
                            href="mailto:lukas@erlang.org"
                            target="_blank">lukas@erlang.org</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> And
                            there it is, conclusive proof that I should
                            not be debugging Rickard's code before
                            lunch. <br>
                            <br>
                            Found the issue, will create a fix for it.
                            As a workaround for R16B you can use a prime
                            number as the number of async threads :)<span><font
                                color="#888888"><br>
                                <br>
                                Lukas</font></span>
                            <div>
                              <div><br>
                                <br>
                                <div>On 13/08/13 10:05, Lukas Larsson
                                  wrote:<br>
                                </div>
                                <blockquote type="cite"> Sigh,
                                  apparently I spoke too soon. <br>
                                  <br>
                                  I remembered incorrectly about the
                                  change. It was in R16B that ErlDrvPort
                                  became a ptr and it was an id before
                                  R16B. Anyways, it is odd that the ptr
                                  is 8 bit aligned on you system. On
                                  mine (Ubuntu 13.04, x86_64) the ptrs
                                  are not aligned and the load is nicely
                                  distributed among async threads. If I
                                  remember correctly you are using
                                  FreeBSD on x86_64? I'll check if I can
                                  reproduce the behavior you are seeing
                                  on our FreeBSD machine. <br>
                                  <br>
                                  Lukas<br>
                                  <br>
                                  <div>On 13/08/13 09:40, Lukas Larsson
                                    wrote:<br>
                                  </div>
                                  <blockquote type="cite"> Hello Rick!<br>
                                    <br>
                                    Which version of Erlang are you
                                    using? From R16B (I think), the
                                    ErlDrvPort datatype no longer is a
                                    pointer to the port struct. Instead
                                    it is the slot id into the port
                                    table and those ids should contain
                                    all values. I did a quick test on my
                                    computer running the latest on maint
                                    on github and seem to get a full
                                    spread over all async threads. <br>
                                    <br>
                                    Lukas<br>
                                    <br>
                                    <div>On 13/08/13 05:40, Rick Reed
                                      wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr">It looks to me as
                                        though there's a bit of a
                                        problem in the way efile_drv.c
                                        generates the
                                        <div>key that's used to select
                                          an async driver queue.  It
                                          uses the address of the port
                                          which</div>
                                        <div>on our system is 8-byte
                                          aligned.  Meanwhile,
                                          erl_async.c does a simple mod
                                          operation</div>
                                        <div>with the number of async
                                          threads, so the number of
                                          threads that can actually be
                                          used</div>
                                        <div>by file operations is 1/8th
                                          of the number configured.  I
                                          suspect this isn't intended.</div>
                                        <div><br>
                                        </div>
                                        <div>Rr</div>
                                        <div><br>
                                        </div>
                                      </div>
                                      <br>
                                      <fieldset></fieldset>
                                      <br>
                                      <pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
                                    </blockquote>
                                    <br>
                                    <br>
                                    <fieldset></fieldset>
                                    <br>
                                    <pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
                                  </blockquote>
                                  <br>
                                  <br>
                                  <fieldset></fieldset>
                                  <br>
                                  <pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                    <br>
                    <fieldset></fieldset>
                    <br>
                    <pre>_______________________________________________
erlang-bugs mailing list
<a moz-do-not-send="true" href="mailto:erlang-bugs@erlang.org" target="_blank">erlang-bugs@erlang.org</a>
<a moz-do-not-send="true" href="http://erlang.org/mailman/listinfo/erlang-bugs" target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a>
</pre>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            _______________________________________________<br>
            erlang-bugs mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:erlang-bugs@erlang.org">erlang-bugs@erlang.org</a><br>
            <a moz-do-not-send="true"
              href="http://erlang.org/mailman/listinfo/erlang-bugs"
              target="_blank">http://erlang.org/mailman/listinfo/erlang-bugs</a><br>
            <br>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>