Logging to one process from thousands: How does it work?

David Hopwood david.nospam.hopwood@REDACTED
Tue Jan 3 21:02:43 CET 2006


> joelr1@REDACTED (Joel Reymont) writes:
> 
>>I have just rewrote some code from Haskell to Erlang and there's one
>>thing that baffles me.
>>
>>I tried to approach Haskell the way I would code in Erlang and set up
>>a logger thread with an unbounded message queue. I then launched a
>>few hundred/thousand threads that traced to the logger periodically.
>>This is no different than using disk_log in Erlang, I think.
>>
>>The Haskell logger thread got quickly overwhelmed with messages I
>>think. The queue build-up was huge. How does it work with Erlang? Is
>>the scheduler specially tuned somehow to give different priorities to
>>different threads? The Haskell scheduler, I believe, is just round-
>>robin.

Raimo Niskanen wrote:
> There is a small fix in the scheduler for the standard
> producer/consumer problem: A process that sends to a
> receiver having a large receive queue gets punished
> with a large reduction (number of function calls)
> count for the send operation, and will therefore
> get smaller scheduling slots.

This makes the problem less likely to occur, but it isn't necessarily enough
to prevent a message backlog in all cases. In principle, the right way to
handle this is to provide back-pressure to the message sources, i.e. to allow
the logging operation to block in each source. The simplest way to do that is
to use bounded queues.

-- 
David Hopwood <david.nospam.hopwood@REDACTED>




More information about the erlang-questions mailing list