[erlang-questions] high-volume logging via a gen_server

[Evans, Matthew]

Another option, and something we have implemented, is to extend this logic by having the log statement actually be a fun.

For example:

logger:funlog(fun() ->
	{"Some message ~p, with data ~p and ~p",
	[Message,Data,?MODULE:evaluate_data(Something)]} end).

This function can squirrel the fun in an ETS table that can be quickly dumped to disc via a separate management process (e.g. ets:to_dets/2 or ets:tab2file/3). The dump process can, if needed, do the evaluation of the fun, or you can even defer that until some later time by reading the dumped data back into the VM (of course, you need to ensure the process that parses the fun has the same module versions as the process that created the fun). Doing it this way, as a fun, means expensive string operations are deferred until later. In effect giving you a form of lazy evaluation.

Matt 


-----Original Message-----
From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Chandru
Sent: Monday, October 04, 2010 8:32 AM
To: djk121@REDACTED
Cc: erlang-questions@REDACTED
Subject: Re: [erlang-questions] high-volume logging via a gen_server

On 4 October 2010 12:57, Dan Kelley <djk121@REDACTED> wrote:

> So, what are good strategies to cope with a large incoming volume of
> messages that all need to wind up in the same logfile?  Is there a more
> efficient way to write to disk than the simple io:format() call than I'm
> using above?  What's a good way to parallelize the logging over multiple
> processes but keep all of the information in one file?
>
>
What we do is to have one public ETS table per log file. All log entries are
written directly to the ETS table by the calling process. Every few seconds,
a dedicated logging process scans the ETS table, accumulates them, and dumps
them to disk in one write operation. This works very well.

cheers
Chandru