[erlang-questions] performance vs msg queue length

[Bernard Duggan]

Ulf Wiger wrote:
> John Erickson wrote:
>   
>> Hello, I have noticed that the amount of work an erlang process gets done
>> seems to decrease as its message queue grows.  In extreme cases, this is
>> understandable, as when the message queue fills up all available memory.
>>  However, even for more moderate cases such as ~100 messages each around
>> 10kB there seems to be significant slowdown.  These are in processes that
>> are not scanning the msg queue; they have a single receive block which is
>> expected to match every message.  Is this expected?
>>     
>
> Are you sure that the work generated by a message doesn't lead to
> e.g. gen_server calls to other processes. Any such call would lead
> to a selective receive operation, which would have to scan the
> message queue.
>   
Just in support of what Ulf is saying, I was thinking exactly the same
thing as you, John, when I was seeing massive performance problems with
big queues (that is "I don't do any selective receives - why is my
performance going to hell?").  Then I realised that the mnesia
transaction operations I was doing internally do selective receives. 
There's no doubt a bunch of other cases too (in addition to the obvious
gen_server:call already mentioned).

> We happened to be discussing this at the SF Erlang Factory (Patrik,
> here's your reminder): in a case such as
>
> call(P, Req) ->
>    MRef = erlang:monitor(process, P),
>    % Mark
>    P ! {'$call', {self(),MRef}, Req},
>    receive
>       {MRef, Reply} -> Reply;
>       {'DOWN', MRef, _, _, Reason} -> erlang:error(Reason)
>    after 5000 ->
>       erlang:error(timeout)
>    end.
>
> the compiler could figure out that it could skip to the
> end of the message queue (the end as of % Mark above),
> since the following receive clause cannot possibly match any
> messages that came in before this point (they all match on
> MRef which is a new and unique object).
>   
Oh, very clever :)  (No sarcasm at all intended).  Presumably that it
could only be safely applied where MRef contained something guaranteed
unique such as that returned by make_ref()?  That would, I'd imagine,
require some minor changes to things like mnesia:transaction() if they
wanted to take advantage of it?

Cheers,

Bernard