[erlang-questions] slow message processing in cluster

[Bob Ippolito]

You might want to take a look at the options in the http module. The
client has some pretty low concurrency by default (e.g. max 2
connections per host, look at max_sessions).

-bob

On 8/14/07, Dustin Sallings <dustin@REDACTED> wrote:
>
>
>  Hi *,
>
>  I've got a cluster of five nodes across two machines and I'm finding my
> message queues are getting really full on my remote nodes.
>
>  I'm pushing some rather large data around.  This application consists of a
> process that reads a file and translates them into http requests (many are
> posts with binary contents), a process to issue the requests, and a process
> to handle the responses.
>
>  I'm using http:request (async) to issue my requests from a process whose
> job is to time and track requests and send the responses back to the
> ``master'' process for dispatch.
>
>  The problem I'm experiencing is in this process.  The source code is
> available here:
>
>
> http://hg.west.spy.net/hg/erlang/httpload/file/fc80675fc138/src/http_requestor.erl
>
>  When this process is running on a remote node, the message queue grows
> fairly rapidly, but it's fine on the local node.  I assumed this might be
> because of the response, so I pushed that out into yet another process
> (locally queue the outbound message so I can quickly get back to my input
> queue), but it had no effect.
>
>  This is what ni() in an interactive session shows me:
>
>
> <5253.14588.0>        net_kernel:spawn_func/6            8024355 23848914
> 7332
> http_requestor        gen:wait_resp_mon/3                     35
>
> <5253.14589.0>        erlang:apply/2                        2584   156940
> 0
> http_response_sender  http_requestor:response_sender_lo
>    1
>  I named the processes for clarity (note that this is showing the version I
> altered to separate local vs. remote node sending -- that detail seemed to
> not matter).
>
>
>  Examining the message queue, I see good mix of request and response
> messages.  I am observing responses coming back from every node, even though
> the queues are always bigger each time I look.
>
>  Any clues as to how I might get stuff moving along a bit more smoothly?
>
>
> --
> Dustin Sallings
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>