[erlang-questions] Thanks for new erlang logger: it is almost perfect

[Max Lapshin]

Hi.

We were using lager for about 7 years and have rewritten it to our own
subsystem called "events" (not opensourced and possible will not be).

Why?

Because logging has changed.

In the ancient times of old sysadmins with beards that are boasting big
uptime, logging was a synonym of appending lines to text file and
extracting data from these lines with a perl script.

Things like webalyzer or radius were an ancient ancestors of modern
ELK/Clickhouse stack.

Nowadays logging is changing and it seems that it has completely splitted
into 2 different data streams.

One data stream is consumed by business analytics. It is collecting events
from happy path. Request started, request finished, upstream request made,
etc.  Here we need to log structured key-value objects that are collected
from all servers of the project and linked together according to some
metadata like "request-id" or "user-id"



Another data stream is about exceptions. They need to be collected and sent
directly to developer tool like Sentry (on-premises and free. It is
excellent) or Airbrake (good old service).



So we have changed from   lager:info("HTTP request handled") to

events:http_request([{ip,IP},{user_id,UserId},....]).

"events" is our own module with small preprocessor that takes all calls
like  events:http_request on compile time and checks if they are defined in
config file.

Thus we can simulate some kind of static typing and give our customers
documentation on all methods without grep.

In config file it is possible to declare sinks like:

notify broken_video_stream {
  only media=mystream;
  sink http://logger-server/logs.php;
}






Then appeared new logger and I see that it has:

1) metadata.  It is very, very important.  Metadata is more important than
text of message.
In our events we have following approach:
events:http_request([{ip,IP},{user_id,UserId},{method,Method}]).

and

{http_request,  ["HTTP request from ip ",ip, {user_id, [" user_id
",user_id],""}, " to ",method]}.

Yes, it is lager conditional formatting and it is really good. Metadata for
robots, text line for sysadmins debugging this software.

2) remote proxy. Very cool.  It happened to be designed exactly 1-1 as we
have designed in Flussonic, so it seems that we have done it right =)

3) lack of central process.

error_logger and lager were a big problem for us because they had single
bottleneck that was working even for debug messages. Very often lager_event
became a reason of crash for flussonic, so we have removed it and now it is
ok.

However, I see that we have a bit more complicated mechanism of log
buffering and handler overload.


https://gist.github.com/maxlapshin/ea632f04f79188aaba46d3a92406f8f6#file-events_router-erl-L94

Before sending message to log handler, we check if it is overloaded, then
check its configuration and only after this we send message to log sink.

Overloading is checked by each sink itself: it sends message to itself:
https://gist.github.com/maxlapshin/7873727e3f93519cf4bb14287264fd3d#file-events_sink-erl-L15


All this is very important, because logging must not kill system due to
slow disk or network.




I want to thank again for new system, I will think how to combine our
private logging subsystem with standard infrastructure.

If our system is of some interest, I can open it code and explain our
design ideas.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20190829/8f43a028/attachment.htm>