[erlang-questions] error_logger and the perils of asynchronicity
Thu May 14 18:58:34 CEST 2009
in all fairness I should say that the system most likely
would have died of other causes, but in the cases I've
seen, error_logger has been among the 3 largest processes
Something similar to the synchronous door send was discussed
by Erik Stenman at EUC02 (http://www.erlang.se/workshop/2002/Stenman.pdf)
I think there are very good reasons to remove the penalty
on send, making ! even more asynchronous. I think that today,
gen_server:call() is so fast that there may not be any need
for a new primitive, but it's doubtful whether gen_event
could be modified to use calls due to BW compatibility reasons.
Michael Radford wrote:
> This seems like just another instance of the "multiple producers, one
> consumer" problem that is easy to get bitten by in Erlang.
> The usual party line response is, if one of your processes is getting
> overloaded like this, you need to implement flow control. But probably
> the OTP team would be reluctant to do that with error_logger because
> most of the time (when messages are rare enough), asynchronous gives
> better performance.
> Another way to address this problem, which I'm sure has been discussed
> before, would be changes to the scheduler.
> What if there were two new send operators, just like !, but with
> scheduling side effects:
> - a "synchronous door" send, for when you are sending a message to a
> server process that will do some work for you and send a reply which you
> will wait for. The scheduling change would be something like: the
> server process is immediately scheduled in for the remainder of the
> client process's time slice, and then the next time the client
> process enters a receive, the server process gets all of the client's
> time slices (as though the client were continuing to run) until it
> sends a message to the client, the client exits the receive, either
> process dies, etc.
> - an "asynchronous door" send, for things like error_logger, and logging in
> general. This would somehow give extra cycles to the server process at
> some point in the future, whether or not the client process still
> exists. Ideally, that would be just enough extra cycles to consume the
> message on average, but the right design is tricky.
> If I understand correctly, right now processes get a scheduling penalty
> for sending to a process with a large message queue (large in bytes or
> messages?). But that doesn't help in all situations, e.g., when new
> processes are being created all the time. (It obviously didn't help in
> Ulf's situation.)
CTO, Erlang Training & Consulting Ltd
More information about the erlang-questions