[erlang-questions] slow message processing in cluster
Wed Aug 15 01:39:30 CEST 2007
You might want to take a look at the options in the http module. The
client has some pretty low concurrency by default (e.g. max 2
connections per host, look at max_sessions).
On 8/14/07, Dustin Sallings <> wrote:
> Hi *,
> I've got a cluster of five nodes across two machines and I'm finding my
> message queues are getting really full on my remote nodes.
> I'm pushing some rather large data around. This application consists of a
> process that reads a file and translates them into http requests (many are
> posts with binary contents), a process to issue the requests, and a process
> to handle the responses.
> I'm using http:request (async) to issue my requests from a process whose
> job is to time and track requests and send the responses back to the
> ``master'' process for dispatch.
> The problem I'm experiencing is in this process. The source code is
> available here:
> When this process is running on a remote node, the message queue grows
> fairly rapidly, but it's fine on the local node. I assumed this might be
> because of the response, so I pushed that out into yet another process
> (locally queue the outbound message so I can quickly get back to my input
> queue), but it had no effect.
> This is what ni() in an interactive session shows me:
> <5253.14588.0> net_kernel:spawn_func/6 8024355 23848914
> http_requestor gen:wait_resp_mon/3 35
> <5253.14589.0> erlang:apply/2 2584 156940
> http_response_sender http_requestor:response_sender_lo
> I named the processes for clarity (note that this is showing the version I
> altered to separate local vs. remote node sending -- that detail seemed to
> not matter).
> Examining the message queue, I see good mix of request and response
> messages. I am observing responses coming back from every node, even though
> the queues are always bigger each time I look.
> Any clues as to how I might get stuff moving along a bit more smoothly?
> Dustin Sallings
> erlang-questions mailing list
More information about the erlang-questions