Logging to one process from thousands: How does it work?

[Sean Hinde]

Hi Chandru,

On 5 Jan 2006, at 16:11, chandru wrote:

> On 03/01/06, Ulf Wiger <ulf@REDACTED> wrote:
>> Den 2006-01-03 22:03:43 skrev Joel Reymont <joelr1@REDACTED>:
>>
>>> How do you implement a bounded queue in Erlang without busy-waiting?
>>
>> Difficult, but you can achieve back-pressure by
>> making the communication with the logger synchronous.
>
> We've had the same problem of multiple processes sending log messages
> to one process. As other people have written, you can do caching
> before writing, use synchronous calls etc. But none of these will
> guarantee any protection. You can still overwhelm a process employing
> all these techniques.

I'm not sure I see this.

It is possible to overwhelm a whole system by sending it too much  
traffic (unless you have some overload protection which can push back  
against incoming traffic - {active, once} ?).

It is also possible to overwhelm the disk subsystem, in which case  
you just need to buy more hardware. Both of these cases are normal  
out of capacity situations which Erlang has shown itself to be more  
than capable of handling.

If the overall system is synchronous you should not see message  
queues fill up out of control.

>
> What will be nice though is support for bounded queues where a process
> can specify the maximum size for it's message queue. New messages
> which arrive once this max size id reached can then be thrown away by
> the runtime system. At the moment, the result of sending a message to
> a process is the message itself(*) - in the case of the recipient
> having a bounded queue, maybe the result can be an error (**) ?

Or make the message consumer drop events which it cannot handle. Just  
receiving a message takes very little time, and then you can  
guarantee that important messages do get handled.

Sean