[erlang-questions] Performance with large queues

[Bernard Duggan]

Jayson Vantuyl wrote:
> Selective receive isn't super-slow if you are picking up messages at the top of the queue.
Indeed, that's what I figured.  In the case I describe, though, the
mnesia messages being looked for will be at the end of the queue - ie
the last thing to be searched.

> It's possible that the process mailbox is collecting messages that never, ever get consumed.
I'm not sure it is, actually, in this case - the gen_server main loop
should always be popping the first message off the queue, whatever it is.

> It so happens, that you can use erlang:process_info(Pid,messages) should give you a copy of the mailbox.
That's an excellent idea - I know, because I had the same one :) When we
first saw this issue I added a monitor to keep an eye on queue lengths,
and indeed to dump the queue once it passed a certain length.  This was
back when it hadn't occurred to me that there was any selective
receiving happening and my best theory was that something was generating
a "storm" of messages.  Sadly, that too has not yet made it to
production.  My newer changes to minimise queue length, of course,
should render this moot unless I've really messed something up.

> Alternatively, does this behavior happen if you restart the service every, say, four hours?
Difficult to say, although in both cases it's taken more than 4 hours to
reach the point of crashing the VM.  The tipping point (presuming that's
what it is) has happened both times under high load, and even then it
runs happily for many days at a time until that point.  I've done some
prodding of the live systems and there's nothing obviously "building up"
 (in terms of memory use, ETS table size, and everything else I could
think to check).  Further, two other systems with similar loads were
running for far longer with no issues.

> Good luck!
Thanks :)

B