[erlang-questions] high-volume logging via a gen_server

[Kaiduan Xie]

If io:format is the bottleneck, you can move the formatting of log to
the caller process, and then dump the formatted string to logger
process.

Also look the following links,

http://blogtrader.net/blog/async_or_sync_log_in
http://blogtrader.net/blog/a_case_study_of_scalable

With erlang, you do not need too much log to analyze the system or to
find bugs, use built-in trace mechanism. If the problem is
re-producible, you can do selective log with the help of seq trace.
For example, you can log the activities of the system for message from
A to B only. This is very powerful to find bugs on a massive loaded
live production system.

Best regards,

/Kaiduan

On Mon, Oct 4, 2010 at 10:35 AM, Evans, Matthew <mevans@REDACTED> wrote:
> Another option, and something we have implemented, is to extend this logic by having the log statement actually be a fun.
>
> For example:
>
> logger:funlog(fun() ->
>        {"Some message ~p, with data ~p and ~p",
>        [Message,Data,?MODULE:evaluate_data(Something)]} end).
>
> This function can squirrel the fun in an ETS table that can be quickly dumped to disc via a separate management process (e.g. ets:to_dets/2 or ets:tab2file/3). The dump process can, if needed, do the evaluation of the fun, or you can even defer that until some later time by reading the dumped data back into the VM (of course, you need to ensure the process that parses the fun has the same module versions as the process that created the fun). Doing it this way, as a fun, means expensive string operations are deferred until later. In effect giving you a form of lazy evaluation.
>
> Matt
>
>
> -----Original Message-----
> From: erlang-questions@REDACTED [mailto:erlang-questions@REDACTED] On Behalf Of Chandru
> Sent: Monday, October 04, 2010 8:32 AM
> To: djk121@REDACTED
> Cc: erlang-questions@REDACTED
> Subject: Re: [erlang-questions] high-volume logging via a gen_server
>
> On 4 October 2010 12:57, Dan Kelley <djk121@REDACTED> wrote:
>
>> So, what are good strategies to cope with a large incoming volume of
>> messages that all need to wind up in the same logfile?  Is there a more
>> efficient way to write to disk than the simple io:format() call than I'm
>> using above?  What's a good way to parallelize the logging over multiple
>> processes but keep all of the information in one file?
>>
>>
> What we do is to have one public ETS table per log file. All log entries are
> written directly to the ETS table by the calling process. Every few seconds,
> a dedicated logging process scans the ETS table, accumulates them, and dumps
> them to disk in one write operation. This works very well.
>
> cheers
> Chandru
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>