[erlang-questions] New OTP app available at GitHub: riak_err

[Ryan Zezeski]

On Thu, Nov 11, 2010 at 8:27 PM, Scott Lystig Fritchie <
fritchie@REDACTED> wrote:

> Max Lapshin <max.lapshin@REDACTED> wrote:
>
> ml> Don't forget to add 5000000 of messages into message queue when
> ml> making these tests =)
>
> Yes, there is that little problem ... but that isn't a problem that
> riak_err is trying to solve(*).  As has been pointed out by Ulf Wiger
> and many others, if there isn't a way to apply backpressure on the
> client (i.e. error/whatever event generators) to get them to slow down,
> then the server (i.e. a gen_event server like error_logger) doesn't have
> many options for defense.
>
> However, on an N-CPU-core system, it's even less likely that one of
> those options will work: throw away work.  Even if the event handler
> could keep state about how many events/sec it could handle and then
> throw events that exceed that limit, there's only 1 CPU core (maximum)
> that can run the gen_event server.  There are N-1 cores still available
> to run obnoxious clients to overflow the server's mailbox anyway.
>
> I'm not above putting a less-vulnerable-to-mailbox-overruns
> gen_event.beam file into another directory, un-sticky the kernel's code
> directory, and load the patched gen_event.beam into a running system
> ... but then again, I'm also known(**) for dirtier-than-my-colleagues'-
> tastes-permit hacks.
>
> -Scott
>
> (*) At least, not yet.
>
> (**) If those individuals don't have better solutions, they are welcome
> to remain silent.  :-)
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>
In would think 5,000,000 msgs on a mailbox would be a sign of a larger
problem and at that point maybe the VM crashing is a good thing?  Then
again, I'm not sure.

I know in my particular case I needed to guard against things like getting a
'badmatch' against a 300MB binary (the binary being a user input).  It's
something that only occurred on bad user input and it shouldn't be something
that causes my app to crash.  Sure, there were things I could have done in
the code to defend against this malformed data, but in some cases it might
make the code more complex and harder to understand.  Furthermore, I can't
possibly anticipate all the potential areas where this could happen without
a _lot_ of discipline.  I know help from some automated tool would go a long
way.  But at the end of the day, patching the error_logger handlers seemed
like the "simplest thing that could possibly work."

While patching the handlers I actually started thinking that maybe this work
should actually go into the io functions themselves.  For example, in
Clojure if you try to print a large data structure in the REPL it will
automatically truncate it unless you dynamically bind a special variable.
 This is something implemented all the way down at the very primitive I/O
level, IIRC.  Remember, you can still shoot yourself in the foot if you're
doing your own I/O.  I know, because I did it myself--ever try to write a
300MB binary as a value in a CouchDB document...I have :)

Scott, I did find it quite the coincidence that you posted this shortly
after my license post :)  I look forward to seeing your solution and
comparing with mine.

-Ryan