<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
I don't think you can rely on the scheduler alone if there are, say
ten producers giving work to one consumer asynchronously. Using
gen_server:call from the producer to the consumer could help here?<br>
<br>
Cheers<br>
<br>
Andy<br>
<br>
On 21/11/2014 12:23, Olivier BOUDEVILLE wrote:<br>
<blockquote
cite="mid:OF8155E6EF.78827D1E-ONC1257D97.00421B20-C1257D97.00441825@notes.edfgdf.fr"
type="cite"><font face="sans-serif" size="2">Hi,</font>
<br>
<br>
<font face="sans-serif" size="2">The reciprocal case (M producers
and
N consumers, with M >> N) may also be a problem. </font>
<br>
<br>
<font face="sans-serif" size="2">I can share a simple program
which seems
to show that the retaliation upon the producers is at least in
some cases
(where M is the number of cores of the machine minus 1, and N=1)
insufficient:
the program will crash and burn, exhausting the memory because
of the mailbox
of the consumer becoming too large. Ideally the back-pressure
would increase
steadily until the mailbox of the producer decreases below
reasonable limits.
Currently, I think that the developer has either to tune M by
hand (which
would depend on the hardware resources...), or to perform some
explicit
yielding (with timer:sleep(1) on the producers). Neither of them
is very
satisfactory.</font>
<br>
<br>
<font face="sans-serif" size="2">More generally, couldn't we hope
that
the scheduling logic could cope with at least most of the M
producers-N
consumers scenarios, without resorting to hints from the
developer like
specifying a process flag or performing explicit yielding? Maybe
(for the
M >> N case) it is just a matter of heavier punishing (ex:
ultimately
exponential) whenever the receiving mailbox becomes too large. </font>
<br>
<br>
<font face="sans-serif" size="2">Best,</font>
<br>
<font face="sans-serif" size="2"><br>
Olivier.<br>
---------------------------<br>
Olivier Boudeville<br>
<br>
EDF R&D : 1, avenue du Général de Gaulle, 92140 Clamart,
France<br>
Département SINETICS, groupe ASICS (I2A), bureau B-226<br>
Office : +33 1 47 65 59 58 / Mobile : +33 6 16 83 37 22 / Fax :
+33 1 47
65 27 13</font>
<br>
<br>
<br>
<table width="100%">
<tbody>
<tr valign="top">
<td width="40%"><font face="sans-serif" size="1"><b><a class="moz-txt-link-abbreviated" href="mailto:darach@gmail.com">darach@gmail.com</a></b>
</font>
<br>
<font face="sans-serif" size="1">Envoyé par :
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions-bounces@erlang.org">erlang-questions-bounces@erlang.org</a></font>
<p><font face="sans-serif" size="1">21/11/2014 11:27</font>
</p>
</td>
<td width="59%">
<table width="100%">
<tbody>
<tr valign="top">
<td>
<div align="right"><font face="sans-serif"
size="1">A</font></div>
</td>
<td><font face="sans-serif" size="1"><a class="moz-txt-link-abbreviated" href="mailto:atill@mail.com">atill@mail.com</a></font>
</td>
</tr>
<tr valign="top">
<td>
<div align="right"><font face="sans-serif"
size="1">cc</font></div>
</td>
<td><font face="sans-serif" size="1"><a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a></font>
</td>
</tr>
<tr valign="top">
<td>
<div align="right"><font face="sans-serif"
size="1">Objet</font></div>
</td>
<td><font face="sans-serif" size="1">Re:
[erlang-questions] Configuring Messaging
Back Pressure</font></td>
</tr>
</tbody>
</table>
<br>
<table>
<tbody>
<tr valign="top">
<td>
<br>
</td>
<td><br>
</td>
</tr>
</tbody>
</table>
<br>
</td>
</tr>
</tbody>
</table>
<br>
<br>
<br>
<font size="3">+1</font>
<p><font size="3">On 21 Nov 2014 04:53, "Andy Till" <</font><a
moz-do-not-send="true" href="mailto:atill@mail.com"><font
color="blue" size="3"><u>atill@mail.com</u></font></a><font
size="3">>
wrote:</font>
<br>
<font size="3">Short outline of how erts_use_sender_punish
works, please
comment if there are any errors in the description.<br>
<br>
erts_use_sender_punish is a flag hard coded to 1 (true), when
it is true
a process sending messages to another process will have its
reduction count
reduced by the number of messages in the receivers message
queue multiplied
by four. Sending messages to processes with zero messages in
the
queue is free in terms of reductions, but sending messages to
load queues
is very expensive and will lead the scheduler to context
switch to another
process more often.<br>
<br>
In the case where the the relationship between of producers
and consumers
is one to one, this makes sense. It provides back pressure
when part
of the system is loaded.<br>
<br>
In the case where there is one producer to many consumers, if
one consumer
gets a flood of messages and the producer is punished then the
punishment
is not only on the producer but on all other consumers because
the producer
cannot create enough work for them as it used all of its
reductions.<br>
<br>
This makes performance unpredictable when work cannot be
spread evenly.<br>
<br>
Could this be made to be configurable using a process flag?<br>
<br>
Cheers<br>
<br>
Andy<br>
<br>
Code:<br>
</font><font color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694"
target="_blank"><font color="blue" size="3"><u>https://github.com/erlang/otp/blob/a70f0ce9b34e4db61dacb8db24f9ab5671ed4c8a/erts/emulator/beam/erl_init.c#L694</u></font></a><font
size="3">
</font><font color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058"
target="_blank"><font color="blue" size="3"><u>https://github.com/erlang/otp/blob/682a6082159568f40615f03d12d44ee70edd14c6/erts/emulator/beam/bif.c#L2058</u></font></a><font
size="3">
</font><font color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504"
target="_blank"><font color="blue" size="3"><u>https://github.com/erlang/otp/blob/a8e12f7168c14cc765a63a51c838d065412795d7/erts/emulator/beam/erl_message.c#L504</u></font></a><font
size="3"><br>
<br>
Previous mentions and issues with this feature:<br>
</font><font color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="http://erlang.org/pipermail/erlang-questions/2011-July/060314.html"
target="_blank"><font color="blue" size="3"><u>http://erlang.org/pipermail/erlang-questions/2011-July/060314.html</u></font></a><font
color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="http://erlang.org/pipermail/erlang-questions/2011-July/060042.html"
target="_blank"><font color="blue" size="3"><u>http://erlang.org/pipermail/erlang-questions/2011-July/060042.html</u></font></a><font
size="3"><br>
<br>
_______________________________________________<br>
erlang-questions mailing list</font><font color="blue"
size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="mailto:erlang-questions@erlang.org" target="_blank"><font
color="blue" size="3"><u>erlang-questions@erlang.org</u></font></a><font
color="blue" size="3"><u><br>
</u></font><a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-questions"
target="_blank"><font color="blue" size="3"><u>http://erlang.org/mailman/listinfo/erlang-questions</u></font></a><tt><font
size="2">_______________________________________________<br>
erlang-questions mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a><br>
</font></tt><a moz-do-not-send="true"
href="http://erlang.org/mailman/listinfo/erlang-questions"><tt><font
size="2">http://erlang.org/mailman/listinfo/erlang-questions</font></tt></a><tt><font
size="2"><br>
</font></tt>
<br>
</p>
<p><br>
Ce message et toutes les pièces jointes (ci-après le 'Message')
sont établis à l'intention exclusive des destinataires et les
informations qui y figurent sont strictement confidentielles.
Toute utilisation de ce Message non conforme à sa destination,
toute diffusion ou toute publication totale ou partielle, est
interdite sauf autorisation expresse.</p>
<p>Si vous n'êtes pas le destinataire de ce Message, il vous est
interdit de le copier, de le faire suivre, de le divulguer ou
d'en utiliser tout ou partie. Si vous avez reçu ce Message par
erreur, merci de le supprimer de votre système, ainsi que toutes
ses copies, et de n'en garder aucune trace sur quelque support
que ce soit. Nous vous remercions également d'en avertir
immédiatement l'expéditeur par retour du message.</p>
<p>Il est impossible de garantir que les communications par
messagerie électronique arrivent en temps utile, sont sécurisées
ou dénuées de toute erreur ou virus.<br>
____________________________________________________</p>
<p>This message and any attachments (the 'Message') are intended
solely for the addressees. The information contained in this
Message is confidential. Any use of information contained in
this Message not in accord with its purpose, any dissemination
or disclosure, either whole or partial, is prohibited except
formal approval.</p>
<p>If you are not the addressee, you may not copy, forward,
disclose or use any part of it. If you have received this
message in error, please delete it and all copies from your
system and notify the sender immediately by return message.</p>
<p>E-mail communication cannot be guaranteed to be timely secure,
error or virus-free.</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
erlang-questions mailing list
<a class="moz-txt-link-abbreviated" href="mailto:erlang-questions@erlang.org">erlang-questions@erlang.org</a>
<a class="moz-txt-link-freetext" href="http://erlang.org/mailman/listinfo/erlang-questions">http://erlang.org/mailman/listinfo/erlang-questions</a>
</pre>
</blockquote>
<br>
</body>
</html>