[erlang-questions] New OTP app available at GitHub: riak_err
Fri Nov 12 03:15:52 CET 2010
On Thu, Nov 11, 2010 at 8:27 PM, Scott Lystig Fritchie <
> Max Lapshin <max.lapshin@REDACTED> wrote:
> ml> Don't forget to add 5000000 of messages into message queue when
> ml> making these tests =)
> Yes, there is that little problem ... but that isn't a problem that
> riak_err is trying to solve(*). As has been pointed out by Ulf Wiger
> and many others, if there isn't a way to apply backpressure on the
> client (i.e. error/whatever event generators) to get them to slow down,
> then the server (i.e. a gen_event server like error_logger) doesn't have
> many options for defense.
> However, on an N-CPU-core system, it's even less likely that one of
> those options will work: throw away work. Even if the event handler
> could keep state about how many events/sec it could handle and then
> throw events that exceed that limit, there's only 1 CPU core (maximum)
> that can run the gen_event server. There are N-1 cores still available
> to run obnoxious clients to overflow the server's mailbox anyway.
> I'm not above putting a less-vulnerable-to-mailbox-overruns
> gen_event.beam file into another directory, un-sticky the kernel's code
> directory, and load the patched gen_event.beam into a running system
> ... but then again, I'm also known(**) for dirtier-than-my-colleagues'-
> tastes-permit hacks.
> (*) At least, not yet.
> (**) If those individuals don't have better solutions, they are welcome
> to remain silent. :-)
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
In would think 5,000,000 msgs on a mailbox would be a sign of a larger
problem and at that point maybe the VM crashing is a good thing? Then
again, I'm not sure.
I know in my particular case I needed to guard against things like getting a
'badmatch' against a 300MB binary (the binary being a user input). It's
something that only occurred on bad user input and it shouldn't be something
that causes my app to crash. Sure, there were things I could have done in
the code to defend against this malformed data, but in some cases it might
make the code more complex and harder to understand. Furthermore, I can't
possibly anticipate all the potential areas where this could happen without
a _lot_ of discipline. I know help from some automated tool would go a long
way. But at the end of the day, patching the error_logger handlers seemed
like the "simplest thing that could possibly work."
While patching the handlers I actually started thinking that maybe this work
should actually go into the io functions themselves. For example, in
Clojure if you try to print a large data structure in the REPL it will
automatically truncate it unless you dynamically bind a special variable.
This is something implemented all the way down at the very primitive I/O
level, IIRC. Remember, you can still shoot yourself in the foot if you're
doing your own I/O. I know, because I did it myself--ever try to write a
300MB binary as a value in a CouchDB document...I have :)
Scott, I did find it quite the coincidence that you posted this shortly
after my license post :) I look forward to seeing your solution and
comparing with mine.
More information about the erlang-questions