Logging to one process from thousands: How does it work?

[chandru]

Hi Sean,

On 05/01/06, Sean Hinde <sean.hinde@REDACTED> wrote:
> Hi Chandru,
>
> On 5 Jan 2006, at 16:11, chandru wrote:
>
> > On 03/01/06, Ulf Wiger <ulf@REDACTED> wrote:
> >> Den 2006-01-03 22:03:43 skrev Joel Reymont <joelr1@REDACTED>:
> >>
> >>> How do you implement a bounded queue in Erlang without busy-waiting?
> >>
> >> Difficult, but you can achieve back-pressure by
> >> making the communication with the logger synchronous.
> >
> > We've had the same problem of multiple processes sending log messages
> > to one process. As other people have written, you can do caching
> > before writing, use synchronous calls etc. But none of these will
> > guarantee any protection. You can still overwhelm a process employing
> > all these techniques.
>
> I'm not sure I see this.
>
> It is possible to overwhelm a whole system by sending it too much
> traffic (unless you have some overload protection which can push back
> against incoming traffic - {active, once} ?).
>
> It is also possible to overwhelm the disk subsystem, in which case
> you just need to buy more hardware. Both of these cases are normal
> out of capacity situations which Erlang has shown itself to be more
> than capable of handling.
>
> If the overall system is synchronous you should not see message
> queues fill up out of control.

Not necessarily. If I decided to allow a 100msgs/sec into the system
and if each of those messages generated a number of log entries, the
message queues in the logger process quickly build up if your hard
disk is not responsive at some point (using synchronous logging). The
problem I have with this is I can't always guarantee that I will
safely handle 100 msgs/sec.

> Or make the message consumer drop events which it cannot handle. Just
> receiving a message takes very little time, and then you can
> guarantee that important messages do get handled.

I suppose I could do this - but that'll mean invoking
process_info(self(), message_queue_len) everytime I handle a new
message. Is the message queue length actually incremented everytime a
message is placed in a process' queue? Or is the length computed
everytime I call process_info/2?

cheers
Chandru