[erlang-questions] Configuring Messaging Back Pressure

[Olivier BOUDEVILLE]

Hi,

The reciprocal case (M producers and N consumers, with M >> N) may also be 
a problem. 

I can share a simple program which seems to show that the retaliation upon 
the producers is at least in some cases (where M is the number of cores of 
the machine minus 1, and N=1) insufficient: the program will crash and 
burn, exhausting the memory because of the mailbox of the consumer 
becoming too large. Ideally the back-pressure would increase steadily 
until the mailbox of the producer decreases below reasonable limits. 
Currently, I think that the developer has either to tune M by hand (which 
would depend on the hardware resources...), or to perform some explicit 
yielding (with timer:sleep(1) on the producers). Neither of them is very 
satisfactory.

More generally, couldn't we hope that the scheduling logic could cope with 
at least most of the M producers-N consumers scenarios, without resorting 
to hints from the developer like specifying a process flag or performing 
explicit yielding? Maybe (for the M >> N case) it is just a matter of 
heavier punishing (ex: ultimately exponential) whenever the receiving 
mailbox becomes too large. 

Best,

Olivier.
---------------------------
Olivier Boudeville

EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
Département SINETICS, groupe ASICS (I2A), bureau B-226
Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 1 47 
65 27 13



darach@REDACTED 
Envoyé par : erlang-questions-bounces@REDACTED
21/11/2014 11:27

A
atill@REDACTED
cc
erlang-questions@REDACTED
Objet
Re: [erlang-questions] Configuring Messaging Back Pressure






+1
On 21 Nov 2014 04:53, "Andy Till" <atill@REDACTED> wrote:
Short outline of how erts_use_sender_punish works, please comment if there 
are any errors in the description.

erts_use_sender_punish is a flag hard coded to 1 (true), when it is true a 
process sending messages to another process will have its reduction count 
reduced by the number of messages in the receivers message queue 
multiplied by four.  Sending messages to processes with zero messages in 
the queue is free in terms of reductions, but sending messages to load 
queues is very expensive and will lead the scheduler to context switch to 
another process more often.

In the case where the the relationship between of producers and consumers 
is one to one, this makes sense.  It provides back pressure when part of 
the system is loaded.

In the case where there is one producer to many consumers, if one consumer 
gets a flood of messages and the producer is punished then the punishment 
is not only on the producer but on all other consumers because the 
producer cannot create enough work for them as it used all of its 
reductions.

This makes performance unpredictable when work cannot be spread evenly.

Could this be made to be configurable using a process flag?

Cheers

Andy

Code:

https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 

https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 

https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504


Previous mentions and issues with this feature:

http://erlang.org/pipermail/erlang-questions/2011-July/060314.html
http://erlang.org/pipermail/erlang-questions/2011-July/060042.html

_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
erlang-questions@REDACTED
http://erlang.org/mailman/listinfo/erlang-questions




Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141121/397dcf6c/attachment.htm>