[erlang-questions] error_logger and the perils of asynchronicity

Scott Lystig Fritchie fritchie@REDACTED
Sat May 16 01:35:09 CEST 2009


BranchingFactor <branchingfactor@REDACTED> wrote:

bf> You could add a maximum-delay configuration option to the log file
bf> system.  As the error messages come in, you timestamp them.  Any
bf> message that sits in the queue for longer than the maximum-delay
bf> will get removed and a message will be printed saying how many such
bf> messages were purged.

This is the same strategy that is advocated in the original SEDA paper
from Brewer et al. (?): if you're overloaded, you need to be able to
figure it out as soon as possible *and* spend as little effort as
possible to drop excess load.

I've done the same thing in a (yet another) key-value database I've been
writing in Erlang.  Each client's request has a wall clock timestamp in
it.  (NTP synchronization is more than good enough.)  If the server
receives a request that's more than X seconds old, it immediately drops
it.  Client app authors don't like it because they can't tell the
difference between server overload and server crash: they'd prefer to
get a "sorry I'm overwhelmed" message instead.

For the error_logger, most messages sent to it are asynchronous anyway.
The major backward compatibility problem would probably be adding the
wallclock time somewhere in the message term.  {shrug}

No one has yet (?) proposed changing the process priority of the
error_logger process.  In an SMP evaluator, even a 'max' process
wouldn't hog-tie an entire VM ... but, then again, with multiple
schedulers, there's more opportunities for active processes to send an
even larger tsunami of async messages to the error_logger, so the effect
is the same.

It wouldn't be a big hack to the VM to add yet another per-process
attribute, maximum mailbox size.  Erlang messaging is "send and pray",
even though coders assume that messaging within the same VM is reliable.
Given the alternative of crashing the entire VM with a memory shortage,
breaking that assumption in the case of the error_logger (and other
carefully-selected, application-specific processes) seems to be an easy
choice: if the mailbox is full, drop the message silently.

-Scott



More information about the erlang-questions mailing list