[erlang-questions] New OTP app available at GitHub: riak_err

[Scott Lystig Fritchie]

Max Lapshin <max.lapshin@REDACTED> wrote:

ml> format("** Generic server ~p terminating \n" ...

ml> Here Msg, State and Reason1 can be horribly big, that is why we
ml> _must_ limit them right here and ...

"Must"?  Hmmm.  That depends.  I agree that it would be a good idea.
But looking at the R13B04 code, that format() call is actually a call to
lib:format(), which sends a message to error_logger with the format
string & args, and then error_logger does the actual formatting.  (I
hope the R14B code doesn't change this pattern....)

If the arguments are big, say X megabytes of heap space, then yes it's
bad that you'll copy X megabytes of stuff onto error_logger's heap via
sending a message.  But X is a much smaller number than the
X*ReallyBadFactor number that we get when io_lib:format() does its naive
thing when formatting the args.

ml> ... and I suppose, that best way is to print into binary inside
ml> failing gen_server because there is one error_logger per all cores
ml> and it may become a bottleneck.

The Hibari project's logging code attempts to do that, since (ignoring
memory for a moment) the formatting is much more CPU-intensive (and can
be done in parallel) than the writing-to-TTY or writing-to-disk stuff
(serialized by gen_event).

I don't know if it would be a positive thing to have the client (i.e.,
the "failing gen_server") use a iolist_to_binary() to flatten the
formatted message into a single binary (which, if larger than 64 bytes)
will be stored in the shared binary heap and thus make the message sent
to error_logger much cheaper.  In theory.  Haven't measured it.

Then there's also the matter of log-level handling, which gets tricky if
the client is making the "should I send this message to error_logger?"
decision.  Assuming a syslog-style logging scheme, if the minimum
logging level is LOG_INFO, then ... the client needs to discover that
logging level before it can decide what to do with a severity LOG_CRIT
or LOG_DEBUG message.  What if The Administrator changes the logging
level?  What if the gen_event server has both syslog-style and
non-syslog-style event handler?  {sigh}  Complexity is, er, um, tough.

-Scott