[erlang-questions] server state output in gen_server error logging

[Matthias Lang]

On Thursday, April 23, Johannes Govaerts wrote:

> I have a gen_server which has binary data in its state (a queue with
> several binaries of around 1500 bytes). When an error occurs,
> logging is flooded with an error report listing all these binaries
> under "Server state".
> 
> Is it possible to configure error logging in a way that server state
> output is limited or disabled?

I don't understand how _any_ production systems can work with the
default error logging setup*. On the other hand, you're only the 
third person I've come across who agrees it's a problem, so maybe
there's a simple solution I'm unaware of.

The solution I use in production: replace the standard error logger
with a custom one which uses an alternative IO formatter to limit the
length of terms when they're printed. Taking that one step at a time:

  1. Enable your own error logger:

        ok = error_logger:add_report_handler(gth_log).

  2. Disable the standard error loggers:

     Assuming you've started Erlang with SASL (i.e. -boot start_sasl):

        gen_event:delete_handler(error_logger, sasl_report_tty_h, unused_arg).
        gen_event:delete_handler(error_logger, error_logger_tty_h, unused_arg).

Your own error logger, i.e. gth_log.erl in step 1, needs to follow the
gen_event behaviour. The important part is gth_log:handle_event, which
could be something like:

  handle_event(Event, State) ->
      io:fwrite("got event ~p\n", [Event]),
      {ok, State}.

Your original problem was that printing out the event from a crashed
gen_server filled up your logs. So you want to avoid io:fwrite in
favour of something which limits the logs to a certain maximum
length. I use trunc_io to do that:

  http://www.corelatus.com/~matthias/trunc_io.erl

it lets you specify an approximate maximum number of characters an error
report may take (and, obviously, it doesn't just use io_lib:fwrite to
print the term and then shorten it...). With that, handle_event becomes
something like:

  handle_event(Event, State) ->
    {String, _} = trunc_io:print(Event, 100),
    io:put_chars(String),
    {ok, State}.

More sophisticated approaches are possible, i.e. you can format
known types of error report in a less ugly way.

Matt

 * i.e. in a crash, arbitrary terms get printed and they can be huge.
   The problems with 'huge' are (a) the logs get filled up, rotating
   out useful information, (b) creating those huge logs can eat huge
   amounts of RAM, so much that on memory-limited systems the VM can
   terminate with out-of-memory and (c) creating those huge logs eats
   huge amounts of CPU, which slows down recovery from a fault.