[erlang-questions] error_logger and the perils of asynchronicity

[Ahmed Ali]

Hi All,

I agree that error_logger, specifically the standard
error_logger_file_h, needs enhancements. I have come across the same
issue with log4erl [1], in which I came across the same issue, and
come to the conclusion that async messages to a primarily IO-bound
process is a bad thing. I thought about adding some kind of persistent
queue as means of communication but it's just easier to use
gen_event:sync_notify.

I don't like the idea of mangling or dropping logging messages
because, having worked with telecom systems, I believe each one is a
view to the system that can be essential. I'd rather to slow down the
system than to mess with them. But, I know many will disagree.

BTW, in the latest log4erl, use sync logging and you can forward all
your error_logger requests to log4erl and get it processed by log4erl.
You can check the latest log4erl in [2].

Regards,

Ahmed

[1] http://log4erl.blogspot.com/2009/04/log4erl-issue-when-high-load.html
[2] http://code.google.com/p/log4erl/

On Fri, May 15, 2009 at 2:07 AM, BranchingFactor
<branchingfactor@REDACTED> wrote:
>
> You could add a maximum-delay configuration option to the log file system.  As the error messages come in, you timestamp them.  Any message that sits in the queue for longer than the maximum-delay will get removed and a message will be printed saying how many such messages were purged.
>
> --- On Thu, 5/14/09, Per Melin <per.melin@REDACTED> wrote:
>
>> From: Per Melin <per.melin@REDACTED>
>> Subject: Re: [erlang-questions] error_logger and the perils of asynchronicity
>> To: sugiyama@REDACTED, erlang-questions@REDACTED
>> Date: Thursday, May 14, 2009, 5:23 PM
>> Marc Sugiyama:
>> > In thinking specifically about error_logger, one
>> solution might be to
>> > have it check its queue and shed work when it is
>> overloaded.  For
>> > example, it could shed work by:
>> >
>> > 1. dumping unformatted messages into the log rather
>> than formatted messages.
>> > 2. abandoning a message and noting in the log that it
>> lost a message.
>> > 3. abandoning messages with no record.
>> >
>> > More generally, some library support for detecting
>> overloaded
>> > processes might be helpful.  I suspect there are
>> several strategies
>> > for doing so (e.g., checking the message queue length.
>> time to process
>> > the message queue, etc.).
>>
>> I'm working on a handler that plugs into error_logger but
>> circumvents
>> some of the issues I've had with error_logger, and also
>> adds some
>> features that I really miss (like date-based log file
>> rotation).
>>
>> I use strategy 2 from your list, throwing away messages but
>> noting it
>> in the log once it catches up. My problem is finding
>> reliable
>> strategies for detecting that we are falling behind.
>>
>> I do the writing to disk in a separate process, and detect
>> flooding by
>> keeping a count of queued up messages waiting to be
>> written. Looking
>> at the size, instead of the length, of the queue would be
>> much better,
>> but I don't know any (fast) way to do that.
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://www.erlang.org/mailman/listinfo/erlang-questions
>>
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>