[erlang-questions] process priority

[Jachym Holecek]

# Mazen Harake 2011-07-04:
> My 2 cents.
> 
> Generally, writing straight to disk is a bad thing. You should have a
> table (ets/mnesia) where you write your log lines and a process that
> periodically flushes it to disk. Depending on your system load this
> will help immensely.

The other way around actually! :) Pass log items to gen_server in synchronous
calls, that gives you end-to-end flow control. Use file in raw mode, leverage
delayed_write flag. Make sure your gen_server has minimal processing overhead.
Any kind of per-item processing in the server is a clear no-go, do all
formatting in caller's context. It helps to do iolist_to_binary/1 in some
carefully chosen callsites -- but you won't need that, following the above
principles will give you a fast-enough solution (sustained load of >50k
messages per second -- iolists about 100B in size -- is no problem, I've seen
something like 70k peak with our logging library). Of course you don't want
to have one logger process for the whole system, instead make it easy for
each application to open as many as it needs for its audit logs, event logs,
trace logs and such.

Using ETS is of course doable, but locking overhead is about the same or
higher than with plain message passing, you lose all flow control thus
risking memory explosion (I have a nice graph handy demonstrating that,
but probably can't publish it), and you copy every messsage twice instead
of just once. (I don't have measurements on the locking and copying stuff,
it's my recollection of reading relevant bits of ERTS -- could be wrong.)

> In very rare cases you would tweak the process priorities and if you
> do you should consider if your solution is "wrong" rather than the
> priority being the bad guy.

I agree with this.

BR,
	-- Jachym