[erlang-questions] error_logger and the perils of asynchronicity

[Ulf Wiger]

in all fairness I should say that the system most likely
would have died of other causes, but in the cases I've
seen, error_logger has been among the 3 largest processes
each time.

Something similar to the synchronous door send was discussed
by Erik Stenman at EUC02 (http://www.erlang.se/workshop/2002/Stenman.pdf)

I think there are very good reasons to remove the penalty
on send, making ! even more asynchronous. I think that today,
gen_server:call() is so fast that there may not be any need
for a new primitive, but it's doubtful whether gen_event
could be modified to use calls due to BW compatibility reasons.

BR,
Ulf W

Michael Radford wrote:
> This seems like just another instance of the "multiple producers, one
> consumer" problem that is easy to get bitten by in Erlang.
> 
> The usual party line response is, if one of your processes is getting
> overloaded like this, you need to implement flow control.  But probably
> the OTP team would be reluctant to do that with error_logger because
> most of the time (when messages are rare enough), asynchronous gives
> better performance.
> 
> Another way to address this problem, which I'm sure has been discussed
> before, would be changes to the scheduler.
> 
> What if there were two new send operators, just like !, but with
> scheduling side effects:
> 
> - a "synchronous door" send, for when you are sending a message to a
>   server process that will do some work for you and send a reply which you
>   will wait for.  The scheduling change would be something like: the
>   server process is immediately scheduled in for the remainder of the
>   client process's time slice, and then the next time the client
>   process enters a receive, the server process gets all of the client's
>   time slices (as though the client were continuing to run) until it
>   sends a message to the client, the client exits the receive, either
>   process dies, etc.
> 
> - an "asynchronous door" send, for things like error_logger, and logging in
>   general.  This would somehow give extra cycles to the server process at
>   some point in the future, whether or not the client process still
>   exists.  Ideally, that would be just enough extra cycles to consume the
>   message on average, but the right design is tricky.
> 
> If I understand correctly, right now processes get a scheduling penalty
> for sending to a process with a large message queue (large in bytes or
> messages?).  But that doesn't help in all situations, e.g., when new
> processes are being created all the time.  (It obviously didn't help in
> Ulf's situation.)


-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com