[erlang-questions] slow message processing in cluster
Tue Aug 14 23:44:52 CEST 2007
I've got a cluster of five nodes across two machines and I'm finding
my message queues are getting really full on my remote nodes.
I'm pushing some rather large data around. This application
consists of a process that reads a file and translates them into http
requests (many are posts with binary contents), a process to issue
the requests, and a process to handle the responses.
I'm using http:request (async) to issue my requests from a process
whose job is to time and track requests and send the responses back
to the ``master'' process for dispatch.
The problem I'm experiencing is in this process. The source code is
When this process is running on a remote node, the message queue
grows fairly rapidly, but it's fine on the local node. I assumed
this might be because of the response, so I pushed that out into yet
another process (locally queue the outbound message so I can quickly
get back to my input queue), but it had no effect.
This is what ni() in an interactive session shows me:
<5253.14588.0> net_kernel:spawn_func/6 8024355
http_requestor gen:wait_resp_mon/3 35
<5253.14589.0> erlang:apply/2 2584
http_response_sender http_requestor:response_sender_lo 1
I named the processes for clarity (note that this is showing the
version I altered to separate local vs. remote node sending -- that
detail seemed to not matter).
Examining the message queue, I see good mix of request and response
messages. I am observing responses coming back from every node, even
though the queues are always bigger each time I look.
Any clues as to how I might get stuff moving along a bit more smoothly?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions