[erlang-bugs] erts_use_sender_punish not always a good thing

[Andy Till]

Short outline of how erts_use_sender_punish works, please comment if 
there are any errors in the description.

erts_use_sender_punish is a flag hard coded to 1 (true), when it is true 
a process sending messages to another process will have its reduction 
count reduced by the number of messages in the receivers message queue 
multiplied by four.  Sending messages to processes with zero messages in 
the queue is free in terms of reductions, but sending messages to load 
queues is very expensive and will lead the scheduler to context switch 
to another process more often.

In the case where the the relationship between of producers and 
consumers is one to one, this makes sense.  It provides back pressure 
when part of the system is loaded.

In the case where there is one producer to many consumers, if one 
consumer gets a flood of messages and the producer is punished then the 
punishment is not only on the producer but on all other consumers 
because the producer cannot create enough work for them as it used all 
of its reductions.

This makes performance unpredictable when work cannot be spread evenly.

Cheers

Andy

Code:

https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 

https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 

https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504