[erlang-questions] error_logger and the perils of asynchronicity

[Christian]

This discussion is going around in circles a bit.

The core problem is that the producers are faster than the consumers.
And the specifics is that the production system is the producer and
the error logger is the consumer.

The problem is not that it is significantly worse to have log messages
lingering in the process mailbox than in an ets table, or in a rabbit
queue. That just pushes the queue buildup to some other place that has
limits and similar failure modes.

While a workaround is to do what you can to speed up the consumer
(more efficient logging format, higher process priority, faster io
etc), the Fix with a capital F, is to wait until the consumer is
ready. Flow control. The producer is bounded by the consumer rate.

If synchronous logging slows down your production system unacceptably,
then you have to ask yourself if you need the logging at all in the
hot spots that were slowed down. Because the alternative is to have
logging bursts potentially explode on you, and take the system down.
Ask yourself what you need the logging for. If you need reliable
logging, you must take the costs for reliable logging. It is as simple
as that.

As usual, a problem in scalability push you all the way back to the
design constraints in the application. If you're doing financial
transaction processing you probably need 100% reliable audit logging,
and cant cheat. For a simple web server is probably okay to
stochastically sample X% of all requests and drop the rest to
extrapolate traffic statistics, catch 404s, and whatever intended use
was. If the logging is "it could be useful to have if we need to debug
things", then event tracing is likely a good substitute, tools like
redbug are designed to be used safely in production.

I could have misunderstood this thread. The error_logger seems to have
some issues in how it behaves under default settings, and some issues
by design. Pointing out those issues so one knows when to use it and
when to roll your own (or to outline feature requests to error_logger)
is of course very valuable for us that didn't actually implement. :-)

On Sat, May 16, 2009 at 14:10, Steve Davis
<steven.charles.davis@REDACTED> wrote:
>
>
> On May 14, 5:55 pm, Oscar Hellström <os...@REDACTED>
> wrote:
>> Ah, but in this case, in the event of a system failure (which results in
>> the VM crashing from running out of memory) we still wouldn't know what
>> should have been logged.
>>
>
> OK, so let's assume you have a complex application that generates lot
> of log messages and you absolutely cannot affort to lose. What about
> logging out to a message queue (e.g. Rabbit)?
>
> /s
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>