[erlang-questions] performance vs msg queue length

[Matthew Sackman]

Hi John, Ulf et al,

On Mon, Apr 19, 2010 at 08:38:12PM +0200, Ulf Wiger wrote:
> John Erickson wrote:
> >Hello, I have noticed that the amount of work an erlang process gets done
> >seems to decrease as its message queue grows.  In extreme cases, this is
> >understandable, as when the message queue fills up all available memory.
> > However, even for more moderate cases such as ~100 messages each around
> >10kB there seems to be significant slowdown.  These are in processes that
> >are not scanning the msg queue; they have a single receive block which is
> >expected to match every message.  Is this expected?
> 
> Are you sure that the work generated by a message doesn't lead to
> e.g. gen_server calls to other processes. Any such call would lead
> to a selective receive operation, which would have to scan the
> message queue.

Right, which is pretty much the reason why we at RabbitMQ extended
gen_server with gen_server2 which, prior to the process_message loop,
completely drains the mailbox, shoving everything into a queue. Messages
are then pulled out of the queue and processed as required. The
advantage is that any :call you subsequently make is likely to result in
a selective receive on a short mailbox, thus it performs *much* better.
http://hg.rabbitmq.com/rabbitmq-server/file/default/src/gen_server2.erl

The things you lose are:
a) deprioritisation of processes which send to an overwhelmed process
b) potentially spending your entire process's CPU time on draining the
mailbox and thus getting no real work done.

There are a number of other enhancements for gen_server2 which we've
added which are documented in the file, but are not really relevant for
this discussion.

Matthew