[erlang-questions] error_logger and the perils of asynchronicity

[BranchingFactor]

You could add a maximum-delay configuration option to the log file system.  As the error messages come in, you timestamp them.  Any message that sits in the queue for longer than the maximum-delay will get removed and a message will be printed saying how many such messages were purged.

--- On Thu, 5/14/09, Per Melin <per.melin@REDACTED> wrote:

> From: Per Melin <per.melin@REDACTED>
> Subject: Re: [erlang-questions] error_logger and the perils of asynchronicity
> To: sugiyama@REDACTED, erlang-questions@REDACTED
> Date: Thursday, May 14, 2009, 5:23 PM
> Marc Sugiyama:
> > In thinking specifically about error_logger, one
> solution might be to
> > have it check its queue and shed work when it is
> overloaded.  For
> > example, it could shed work by:
> >
> > 1. dumping unformatted messages into the log rather
> than formatted messages.
> > 2. abandoning a message and noting in the log that it
> lost a message.
> > 3. abandoning messages with no record.
> >
> > More generally, some library support for detecting
> overloaded
> > processes might be helpful.  I suspect there are
> several strategies
> > for doing so (e.g., checking the message queue length.
> time to process
> > the message queue, etc.).
> 
> I'm working on a handler that plugs into error_logger but
> circumvents
> some of the issues I've had with error_logger, and also
> adds some
> features that I really miss (like date-based log file
> rotation).
> 
> I use strategy 2 from your list, throwing away messages but
> noting it
> in the log once it catches up. My problem is finding
> reliable
> strategies for detecting that we are falling behind.
> 
> I do the writing to disk in a separate process, and detect
> flooding by
> keeping a count of queued up messages waiting to be
> written. Looking
> at the size, instead of the length, of the queue would be
> much better,
> but I don't know any (fast) way to do that.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>