[erlang-questions] Sender punishment removed

[Karl Nilsson]

Thanks Jesper, keeping flow control to the input edge makes perfect sense
to me but developing a good feedback mechanism that is both safe and not
overly cautious is likely to be quite challenging in non-trivial,
potentially distributed domains.

I have some thinking to do. :) Any further reading along these lines that
anyone knows of would be very welcome.

On Fri, 19 Jan 2018 at 21:17 Jesper Louis Andersen <
jesper.louis.andersen@REDACTED> wrote:

> As Lukas already wrote on slack somewhere:
>
> Imagine you have a 2000'es machine: single core, around 400 mhz if you are
> lucky. In that setting, sender punishment can in some situations rectify a
> system that is going overboard. We simply let the offending process have
> less time on the scheduler in the hope that the overloaded mailbox process
> can catch up and do its work. It is not a surefire solution, but it may
> avoid some situations in which the system would otherwise topple.
>
> Fast forward 18 years. Now, the machines are multicore, at least 4 threads
> commonly. Here, a sender might live on one core whereas the reciever might
> live on another process. It is less clear why the punishment strategy is
> good: we get to stop the sender, but there were already a scheduler for the
> other core and it is still overloaded. Worse, perhaps all the other cores
> are able to send messages through to the overloaded process.
>
> As for the flow control: Erlang systems already employ flow control,
> namely TCP flow control between distributed nodes. I've seen two recent
> problems pertaining to having flow control inside flow control: gRPC has 3
> layers: gRPC itself, HTTP/2 and TCP. And HTTP/2 has a layer on top of TCP.
> This is dangerous as the flow control of the underlying system can
> interfere with the flow control of the system above.
>
> By extension, any Erlang-mechanism of flow control needs to protect
> against a scenario where your application has its own layer and make sure
> it doesn't interfere.
>
> Personally, I think Ulf Wiger's "Jobs" model paved the way[0]: Apply flow
> control on input edge of the system, but don't apply it internally. If you
> do this correct, then a system shouldn't overload because of the
> border-limit. If you apply internal flow control, you also expose yourself
> to the danger of an artificial internal bottleneck. Rather, sample
> internally and use this as a feedback mechanism for the input edge.
>
> Also note distributed flow control is a considerably harder problem to
> solve, and since Erlang is distributed by default, any general solution has
> to address this as well.
>
> [0] https://github.com/uwiger/jobs/blob/master/doc/erlang07g-wiger.pdf
>
> On Fri, Jan 19, 2018 at 9:51 AM Karl Nilsson <kjnilsson@REDACTED> wrote:
>
>> So I saw that the sender punishment was removed in [1]. The commit
>> message doesn't outline any of the reasoning behind this. Is there any more
>> details available about this anywhere I can read? I understand it never
>> really worked that well but it would still be interesting to understand a
>> bit further.
>>
>> On a similar note what is the current thinking on flow control between
>> erlang processes? Are there any improvements on mixing in a few calls in
>> with the casts?
>>
>> [1]
>> https://github.com/erlang/otp/commit/2e601a2efc19d64ed0628a5973596e6331ddcc7c
>>
> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180122/189623db/attachment.htm>