[erlang-questions] error_logger and the perils of asynchronicity
Thu May 14 21:16:33 CEST 2009
In thinking specifically about error_logger, one solution might be to
have it check its queue and shed work when it is overloaded. For
example, it could shed work by:
1. dumping unformatted messages into the log rather than formatted messages.
2. abandoning a message and noting in the log that it lost a message.
3. abandoning messages with no record.
More generally, some library support for detecting overloaded
processes might be helpful. I suspect there are several strategies
for doing so (e.g., checking the message queue length. time to process
the message queue, etc.).
On Thu, May 14, 2009 at 9:58 AM, Ulf Wiger
> in all fairness I should say that the system most likely
> would have died of other causes, but in the cases I've
> seen, error_logger has been among the 3 largest processes
> each time.
> Something similar to the synchronous door send was discussed
> by Erik Stenman at EUC02 (http://www.erlang.se/workshop/2002/Stenman.pdf)
> I think there are very good reasons to remove the penalty
> on send, making ! even more asynchronous. I think that today,
> gen_server:call() is so fast that there may not be any need
> for a new primitive, but it's doubtful whether gen_event
> could be modified to use calls due to BW compatibility reasons.
> Ulf W
> Michael Radford wrote:
>> This seems like just another instance of the "multiple producers, one
>> consumer" problem that is easy to get bitten by in Erlang.
>> The usual party line response is, if one of your processes is getting
>> overloaded like this, you need to implement flow control. But probably
>> the OTP team would be reluctant to do that with error_logger because
>> most of the time (when messages are rare enough), asynchronous gives
>> better performance.
>> Another way to address this problem, which I'm sure has been discussed
>> before, would be changes to the scheduler.
>> What if there were two new send operators, just like !, but with
>> scheduling side effects:
>> - a "synchronous door" send, for when you are sending a message to a
>> server process that will do some work for you and send a reply which you
>> will wait for. The scheduling change would be something like: the
>> server process is immediately scheduled in for the remainder of the
>> client process's time slice, and then the next time the client
>> process enters a receive, the server process gets all of the client's
>> time slices (as though the client were continuing to run) until it
>> sends a message to the client, the client exits the receive, either
>> process dies, etc.
>> - an "asynchronous door" send, for things like error_logger, and logging in
>> general. This would somehow give extra cycles to the server process at
>> some point in the future, whether or not the client process still
>> exists. Ideally, that would be just enough extra cycles to consume the
>> message on average, but the right design is tricky.
>> If I understand correctly, right now processes get a scheduling penalty
>> for sending to a process with a large message queue (large in bytes or
>> messages?). But that doesn't help in all situations, e.g., when new
>> processes are being created all the time. (It obviously didn't help in
>> Ulf's situation.)
> Ulf Wiger
> CTO, Erlang Training & Consulting Ltd
> erlang-questions mailing list
More information about the erlang-questions