[erlang-questions] Configuring Messaging Back Pressure

Mon Nov 24 10:53:50 CET 2014

Hi,

> I don't think you can rely on the scheduler alone if there are, say 
> ten producers giving work to one consumer asynchronously.  Using 
> gen_server:call from the producer to the consumer could help here?

Indeed, using synchronous operations (not necessarily involving OTP) 
between the producer(s) and the consumer(s) would balance the processings, 
at the expense of many more messages being sent. We could imagine sending 
one "ack" from the consumer(s) only every P messages received (a bit like 
a TCP window), but the logic would then be more complex to manage. 

On the other hand, if the scheduler could "naturally" slow down parts of 
the chain to make the work of loaded processes easier, this would be neat. 
This is a bit surprising that, even with a reduction count reduced by the 
number of messages in the receivers message queue multiplied by four, 
producers are still scheduled at least relatively often. Maybe the VM 
prefers to keep all cores busy even if it results in having low priority 
processes keeping on feeding already loaded processes. Maybe it shouldn't, 
especially if prefering stability over efficiency! (i.e. priority #1: do 
not saturate loaded processes; then only, priority #2: try to use all 
cores) 

Best,

Olivier.

> Cheers
> 
> Andy
> 
> On 21/11/2014 12:23, Olivier BOUDEVILLE wrote:
> Hi, 
> 
> The reciprocal case (M producers and N consumers, with M >> N) may 
> also be a problem. 
> 
> I can share a simple program which seems to show that the 
> retaliation upon the producers is at least in some cases (where M is
> the number of cores of the machine minus 1, and N=1) insufficient: 
> the program will crash and burn, exhausting the memory because of 
> the mailbox of the consumer becoming too large. Ideally the back-
> pressure would increase steadily until the mailbox of the producer 
> decreases below reasonable limits. Currently, I think that the 
> developer has either to tune M by hand (which would depend on the 
> hardware resources...), or to perform some explicit yielding (with 
> timer:sleep(1) on the producers). Neither of them is very satisfactory. 
> 
> More generally, couldn't we hope that the scheduling logic could 
> cope with at least most of the M producers-N consumers scenarios, 
> without resorting to hints from the developer like specifying a 
> process flag or performing explicit yielding? Maybe (for the M >> N 
> case) it is just a matter of heavier punishing (ex: ultimately 
> exponential) whenever the receiving mailbox becomes too large. 
> 
> Best, 
> 
> Olivier.
> ---------------------------
> Olivier Boudeville
> 
> EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart, France
> Département SINETICS, groupe ASICS (I2A), bureau B-226
> Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax : +33 
> 1 47 65 27 13 
> 

> 
> darach@REDACTED 
> Envoyé par : erlang-questions-bounces@REDACTED 
> 21/11/2014 11:27 
> 
> A
> 
> atill@REDACTED 
> 
> cc
> 
> erlang-questions@REDACTED 
> 
> Objet
> 
> Re: [erlang-questions] Configuring Messaging Back Pressure
> 
> 
> 
> 
> +1 
> On 21 Nov 2014 04:53, "Andy Till" <atill@REDACTED> wrote: 
> Short outline of how erts_use_sender_punish works, please comment if
> there are any errors in the description.
> 
> erts_use_sender_punish is a flag hard coded to 1 (true), when it is 
> true a process sending messages to another process will have its 
> reduction count reduced by the number of messages in the receivers 
> message queue multiplied by four.  Sending messages to processes 
> with zero messages in the queue is free in terms of reductions, but 
> sending messages to load queues is very expensive and will lead the 
> scheduler to context switch to another process more often.
> 
> In the case where the the relationship between of producers and 
> consumers is one to one, this makes sense.  It provides back 
> pressure when part of the system is loaded.
> 
> In the case where there is one producer to many consumers, if one 
> consumer gets a flood of messages and the producer is punished then 
> the punishment is not only on the producer but on all other 
> consumers because the producer cannot create enough work for them as
> it used all of its reductions.
> 
> This makes performance unpredictable when work cannot be spread evenly.
> 
> Could this be made to be configurable using a process flag?
> 
> Cheers
> 
> Andy
> 
> Code:
> 
> https://github.com/erlang/otp/blob/
> 
a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694 

> https://github.com/erlang/otp/blob/
> 682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058 
> https://github.com/erlang/otp/blob/
> 
a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504
> 
> Previous mentions and issues with this feature:
> 
> http://erlang.org/pipermail/erlang-questions/2011-July/060314.html
> http://erlang.org/pipermail/erlang-questions/2011-July/060042.html
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

> 
> Ce message et toutes les pièces jointes (ci-après le 'Message') sont
> établis à l'intention exclusive des destinataires et les 
> informations qui y figurent sont strictement confidentielles. Toute 
> utilisation de ce Message non conforme à sa destination, toute 
> diffusion ou toute publication totale ou partielle, est interdite 
> sauf autorisation expresse.
> Si vous n'êtes pas le destinataire de ce Message, il vous est 
> interdit de le copier, de le faire suivre, de le divulguer ou d'en 
> utiliser tout ou partie. Si vous avez reçu ce Message par erreur, 
> merci de le supprimer de votre système, ainsi que toutes ses copies,
> et de n'en garder aucune trace sur quelque support que ce soit. Nous
> vous remercions également d'en avertir immédiatement l'expéditeur 
> par retour du message.
> Il est impossible de garantir que les communications par messagerie 
> électronique arrivent en temps utile, sont sécurisées ou dénuées de 
> toute erreur ou virus.
> ____________________________________________________
> This message and any attachments (the 'Message') are intended solely
> for the addressees. The information contained in this Message is 
> confidential. Any use of information contained in this Message not 
> in accord with its purpose, any dissemination or disclosure, either 
> whole or partial, is prohibited except formal approval.
> If you are not the addressee, you may not copy, forward, disclose or
> use any part of it. If you have received this message in error, 
> please delete it and all copies from your system and notify the 
> sender immediately by return message.
> E-mail communication cannot be guaranteed to be timely secure, error
> or virus-free.
> 

> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

Ce message et toutes les pièces jointes (ci-après le 'Message') sont établis à l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme à sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse.

Si vous n'êtes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez reçu ce Message par erreur, merci de le supprimer de votre système, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions également d'en avertir immédiatement l'expéditeur par retour du message.

Il est impossible de garantir que les communications par messagerie électronique arrivent en temps utile, sont sécurisées ou dénuées de toute erreur ou virus.
____________________________________________________

This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval.

If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message.

E-mail communication cannot be guaranteed to be timely secure, error or virus-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20141124/72708b3a/attachment.htm>