Logger under load

[Michael Uvarov]

Hi,

Sync/async logging was a common issue with lager too. I highly suggest not
to use sync logging to avoid extra timeout issues which is pretty hard to
debug.

Dropping the whole sinks (the whole pending logger message queue) was an
option for lager.

Better option would be to count how many events are logged from each
location and start skipping log events from too chatty locations (and log
how many events were dropped accordingly). But I haven't seen such "smart"
code anywhere yet.

On Fri, 7 Aug 2020, 10:28 Steve Strong, <steve@REDACTED> wrote:

> I think a child process to perform the synchronous logging would work fine
> - something like:
>
> child(Server) ->
>   receive
>     {do_log, Ref, From, Msg} ->
>       try
>         Res = gen_server:call(Server, Msg),
>         From ! {Ref, {completed, Res}}
>       catch
>         _:{timeout,_} ->
>           From ! {Ref, timeout}
>       end;
>
>     {Ref, ok} ->
>       ok;
>
>     {Ref, dropped} ->
>       ok
>   end,
>   child(Server).
>
> (I doubt that even compiles, but should give the idea!) - when logger_olp
> is in sync mode, instead of it doing the gen_server:call, it sends a do_log
> message to the child and then waits for the response, which it is
> guaranteed to get.  Any timeouts are caught in the child, and any resulting
> stray replies will also be getting sent to the child where they can be
> dropped
>
> > On 7 Aug 2020, at 09:14, Łukasz Niemier <lukasz@REDACTED> wrote:
> >
> >> In the event of a timeout, it is handling the exception dropping the
> log request.  And it’s when this occurs that there’s a problem - the
> gen_server call to the handler is obviously still in operation, even though
> the caller (logger, in this case) has timed out and moved on.  At some
> point it completes, and a message of {Ref, ok} or {Ref, dropped} gets
> returned to the caller.
> >
> > I am working on the "public" version of logger_olp so I can check out
> that, but I do not think that much could be done there except of just
> handling and dropping the stray messages in the handling process itself as
> in one way or another there will be need to fetch the state of the process.
> So in the end, there always be some call that can timeout and will result
> in stray messages.
> >
> > --
> >
> > Łukasz Niemier
> > lukasz@REDACTED
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20200807/4ef6ed26/attachment.htm>