<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Hi *,<br></div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I've got a cluster of five nodes across two machines and I'm finding my message queues are getting really full on my remote nodes.<br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I'm pushing some rather large data around. This application consists of a process that reads a file and translates them into http requests (many are posts with binary contents), a process to issue the requests, and a process to handle the responses.<br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I'm using <a href="http:request">http:request</a> (async) to issue my requests from a process whose job is to time and track requests and send the responses back to the ``master'' process for dispatch.<br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>The problem I'm experiencing is in this process. The source code is available here:<br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: 12.0px Helvetica"><a href="http://hg.west.spy.net/hg/erlang/httpload/file/fc80675fc138/src/http_requestor.erl">http://hg.west.spy.net/hg/erlang/httpload/file/fc80675fc138/src/http_requestor.erl</a></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-tab-span" style="white-space:pre"> </span>When this process is running on a remote node, the message queue grows fairly rapidly, but it's fine on the local node. I assumed this might be because of the response, so I pushed that out into yet another process (locally queue the outbound message so I can quickly get back to my input queue), but it had no effect.<br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-tab-span" style="white-space:pre"> </span>This is what ni() in an interactive session shows me:<br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><div><5253.14588.0> net_kernel:spawn_func/6 8024355 23848914 7332</div><div>http_requestor gen:wait_resp_mon/3 35 </div><div><5253.14589.0> erlang:apply/2 2584 156940 0</div><div>http_response_sender http_requestor:response_sender_lo 1 </div><br class="webkit-block-placeholder"></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span class="Apple-tab-span" style="white-space:pre"> </span>I named the processes for clarity (note that this is showing the version I altered to separate local vs. remote node sending -- that detail seemed to not matter).<br class="webkit-block-placeholder"></div> </div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Examining the message queue, I see good mix of request and response messages. I am observing responses coming back from every node, even though the queues are always bigger each time I look.<br class="webkit-block-placeholder"></div><div><br class="webkit-block-placeholder"></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Any clues as to how I might get stuff moving along a bit more smoothly?<br class="webkit-block-placeholder"></div><br><div> <span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div>-- </div><div>Dustin Sallings</div><br class="Apple-interchange-newline"></span> </div><br></body></html>