[erlang-questions] Sender punishment removed

[Fred Hebert]

I have written Handling Overload (https://ferd.ca/handling-overload.html)
as tour of the multiple options in Erlang. It may prove helpful.

On Mon, Jan 22, 2018 at 4:52 AM, Karl Nilsson <kjnilsson@REDACTED> wrote:

> Thanks Jesper, keeping flow control to the input edge makes perfect sense
> to me but developing a good feedback mechanism that is both safe and not
> overly cautious is likely to be quite challenging in non-trivial,
> potentially distributed domains.
>
> I have some thinking to do. :) Any further reading along these lines that
> anyone knows of would be very welcome.
>
> On Fri, 19 Jan 2018 at 21:17 Jesper Louis Andersen <
> jesper.louis.andersen@REDACTED> wrote:
>
>> As Lukas already wrote on slack somewhere:
>>
>> Imagine you have a 2000'es machine: single core, around 400 mhz if you
>> are lucky. In that setting, sender punishment can in some situations
>> rectify a system that is going overboard. We simply let the offending
>> process have less time on the scheduler in the hope that the overloaded
>> mailbox process can catch up and do its work. It is not a surefire
>> solution, but it may avoid some situations in which the system would
>> otherwise topple.
>>
>> Fast forward 18 years. Now, the machines are multicore, at least 4
>> threads commonly. Here, a sender might live on one core whereas the
>> reciever might live on another process. It is less clear why the punishment
>> strategy is good: we get to stop the sender, but there were already a
>> scheduler for the other core and it is still overloaded. Worse, perhaps all
>> the other cores are able to send messages through to the overloaded process.
>>
>> As for the flow control: Erlang systems already employ flow control,
>> namely TCP flow control between distributed nodes. I've seen two recent
>> problems pertaining to having flow control inside flow control: gRPC has 3
>> layers: gRPC itself, HTTP/2 and TCP. And HTTP/2 has a layer on top of TCP.
>> This is dangerous as the flow control of the underlying system can
>> interfere with the flow control of the system above.
>>
>> By extension, any Erlang-mechanism of flow control needs to protect
>> against a scenario where your application has its own layer and make sure
>> it doesn't interfere.
>>
>> Personally, I think Ulf Wiger's "Jobs" model paved the way[0]: Apply flow
>> control on input edge of the system, but don't apply it internally. If you
>> do this correct, then a system shouldn't overload because of the
>> border-limit. If you apply internal flow control, you also expose yourself
>> to the danger of an artificial internal bottleneck. Rather, sample
>> internally and use this as a feedback mechanism for the input edge.
>>
>> Also note distributed flow control is a considerably harder problem to
>> solve, and since Erlang is distributed by default, any general solution has
>> to address this as well.
>>
>> [0] https://github.com/uwiger/jobs/blob/master/doc/erlang07g-wiger.pdf
>>
>> On Fri, Jan 19, 2018 at 9:51 AM Karl Nilsson <kjnilsson@REDACTED> wrote:
>>
>>> So I saw that the sender punishment was removed in [1]. The commit
>>> message doesn't outline any of the reasoning behind this. Is there any more
>>> details available about this anywhere I can read? I understand it never
>>> really worked that well but it would still be interesting to understand a
>>> bit further.
>>>
>>> On a similar note what is the current thinking on flow control between
>>> erlang processes? Are there any improvements on mixing in a few calls in
>>> with the casts?
>>>
>>> [1] https://github.com/erlang/otp/commit/2e601a2efc19d64ed0628a5973596e
>>> 6331ddcc7c
>>>
>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180122/e8bc0b48/attachment.htm>