[erlang-questions] Reliable kernels?

[Richard A. O'Keefe]

I was introduced to the concept of a "million-year bug" today.
That's a bug in a program that, if you were running it on a
single CPU, would be expected to show up once in a million years.

With a couple of thousand million Linux kernels around the world
in phones &c, we can expect a million year bug in the kernel to
show up tens of times a day.

There's been a spate of problems with 911 in Dallas being overloaded
with bogus calls apparently sent autonomously by certain mobile
phones, to the point where a man and a child are thought to have
died because genuine 911 callers were put on hold for a long time.
That's probably *not* a million-year bug, but a million-year bug
might do that kind of thing.

Now me, I'm still happily pottering away on 4-core and 16-core
machines (even a 1-core machine that sees a lot of use because it
Just Keeps Working).  But I'm talking to people who want to use
unbelievable amounts of computing power.

I understand the Erlang Way:  write your software as lots of
small things communicating through narrow protocols, *expect*
failure and deal with it.  I believe!  Praise Joe, I believe!

That's not the way the people I'm talking to think.  They've got
a somewhat resilient data flow scheme they're proud of that has
thousands and tens of thousands of nodes hooked up through
Python, where the protocol between the nodes is Pyro.  Not,
"uses Pyro", "IS Pyro".

I'm supposed to tell these people what would be a good stripped
down kernel to use (or what would be a good kernel to strip
further), and I'm tempted to start by saying "strip away Python"
which certainly won't make me popular (;-).  But thinking about
error rates and million-year bugs has me thinking harder.

It seems as if we have to think in terms of *expecting* the
kernel itself to be unreliable-at-scale, so that something like
JailHouse might be the right level to start.

Does anyone have any experience with trying to harden a large system
against faults in the distribution layer and in the kernel, and have
any advice they'd care to share?

None of this is Erlang-specific, it's just that I think the Erlang
community are likely to have more relevant experience than most.