[erlang-questions] Processes & Fault Tolerance

Tue Jan 4 10:52:50 CET 2011

On Mon, Jan 3, 2011 at 5:30 PM, Edmond Begumisa
<ebegumisa@REDACTED> wrote:
>> Let's start with how we do error recovery.
>>
>> Imagine two linked processes A and B on separate machines. A is the
>> master process.
>> B is a passive processes that will take over if A fails.
>>
>> A sends a stream of state update messages S1, S2, S3,... to B. The
>> state messages contain enough information
>> for B to do whatever A was doing should A fail. If A fails B will
>> receive an EXIT signal.
>>
>> If B does receive an exit signal it knows A has failed, so it carries
>> on doing whatever A was doing
>> using the information it last received in a state update message.
>>
>> That's it and all about it - nothing to do with supervisors etc,
>
> Ooooohh! THANK YOU, THANK YOU, THANK YOU.

Thanks.

Another problem in understand is that we assume the meaning of words.

Once upon a time I was talking to a guy and used the word "fault tolerance"
we talked for hours and thought we understood what this word meant. Then
he said something that implied he was building a fault-tolerant system
on one computer.
I said it was impossible. He said, "you catch and handle all the exceptions .."

I said, "The entire computer might crash"

He said, "Oh"

There was a long silence.

I told him about Erlang ...

/Joe

>
> *Now* I get it. This is precisely what I was trying to understand. The
> missing link was sending redundant messages. It all makes sense now -- it's
> so simple. All I have to do to get fault tolerance at the process level is
> have a group of N redundant processes waiting for exit-signals and forward
> state changes to N-1 of them. I could even send to N/2 and have some decent
> tolerance at the expense of consistency of the nodes.
>
> To preemptively answer the question I sent Ulf and Mazen... for that
> distributed database, Process B would be set up to also receive read
> requests forwarded from Process A but not respond to them unless it gets an
> exit signal it can understand like disk_fail_error. Any changes in state
> would also be forwarded to existing redundant processes.
>
> Thanks everyone for your patience in helping me understand fault-tolerance.
> I've already started thinking about some cool things I can do with this in
> my program.
>
> - Edmond -
>
>
>
> On Tue, 04 Jan 2011 00:36:30 +1100, Joe Armstrong <erlang@REDACTED> wrote:
>
>> I think you are getting confused between OTP supervisors etc. and the
>> general notion of "take-over"
>>
>> Let's start with how we do error recovery.
>>
>> Imagine two linked processes A and B on separate machines. A is the
>> master process.
>> B is a passive processes that will take over if A fails.
>>
>> A sends a stream of state update messages S1, S2, S3,... to B. The
>> state messages contain enough information
>> for B to do whatever A was doing should A fail. If A fails B will
>> receive an EXIT signal.
>>
>> If B does receive an exit signal it knows A has failed, so it carries
>> on doing whatever A was doing
>> using the information it last received in a state update message.
>>
>> That's it and all about it - nothing to do with supervisors etc,
>> <aside> This is how we do things in a distributed system, obviously
>> real fault tolerance needs more than one machine to cover the case
>> where an entire machine fails. This is also why we copy everything, if
>> an entire machine has failed and we have a pointer to vital data on
>> that machine then we are in big trouble.
>>
>> If it works like this in the distributed case then we reasoned it
>> should work exactly the same way when both processes are on the same
>> machine. Why is this? - because it would be very difficult to program
>> if the primitives and mechanisms involved were completely different
>> when the remote process lay on the same machine as the process it was
>> talking to or on a different machine.
>>
>> It might be that we actually handle these two cases differently in the
>> VM, but as far as the programmer is concerned the system should behave
>> the same way, and the only observable change should
>> concerned with latencies.
>> </aside>
>>
>> To make the above possible we need a couple of primitives in Erlang,
>> namely process_flag trap_exits, and spawn_link. That's all you need.
>>
>> The basic system properties you need to make a fault-tolerant systems
>> are the abilities to detect remote errors and send and receive messages.
>>
>> What the OTP supervisors etc. do are build layers of abstraction on
>> top of spawn-link and trap_exits
>> in order to simplify programming common use-cases.
>>
>> Fault detection and recovery is not arbitrary or magic in any sense,
>> It has to be part of the system architecture like everything else. One
>> common fault in making systems is to specify and design for the normal
>> behavior
>> but not specify and design how fault-detection and correction work.
>> This is probably due to a legacy of
>> programming in languages that have very limited ability to detect
>> errors and recover from them.
>>
>> A C programmer with a single thread of control will take extreme
>> measures to avoid crashing their program
>> they have to,  there is only one thread of control. Given multiple
>> threads of control and the ability to remotely
>> detect errors your entire way of thinking changes and the emphasis
>> changes from thinking "how to I avoid a crash"
>> to thinking "given that a crash has occurred and been detected, how to
>> I fix things up and carry on"
>>
>> The Erlang philosophy is that "things will crash", so we have to
>> detect the crashes, fix the bit that broke and
>> carry on - we do not make excessive efforts to avoid crashes, but we
>> do try very hard to repair things after they have gone wrong.
>>
>> Fixing things after they have broken is often easier than preventing
>> them from breaking. Preventing them
>> from breaking is not theoretically possible - we can't even prove
>> simple things, like that a program will terminate
>> (the famous halting problem) - but we can easily detect failures and
>> put the system to rights by applying
>> certain invariants.
>>
>> OTP supervisors are just trees of processes with certain restart rules
>> that experience has shown us
>> to be useful.
>>
>> The main value of supervisors etc. (and of all the OTP behaviors) is
>> in building large systems with many
>> programmers involved. Rather than inventing their own patterns, the
>> programmers reuse a common set of
>> patterns, which makes the projects easier to manage.
>>
>> /Joe
>>
>>
>>
>>
>> On Mon, Jan 3, 2011 at 6:21 AM, Edmond Begumisa
>> <ebegumisa@REDACTED> wrote:
>>>
>>> Thanks for your response.
>>>
>>> Firstly, let me make my question a little clearer...
>>>
>>> To rephrase: For processes, "share nothing for the sake of concurrency" -
>>> I
>>> get, both in concept and application. "Share nothing for the sake of
>>> fault-tolerance" - I get in concept but not in application.
>>>
>>> Yet as I understand it, it is for the latter reason Erlang shares
>>> nothing*
>>> and not the former. Interpretation: I must be completely missing the
>>> point
>>> in regards to Erlang processes and sharing nothing. This is what I want
>>> to
>>> understand in application. In addition to the "side" effect of sane
>>> concurrency (which coming from a chaotic multi-threading shared-memory
>>> world
>>> I fully appreciate and practically make use of everyday), how can I also
>>> make use of the "real" reason Erlang processes share nothing -- fault
>>> tolerance? Practically/illustratively speaking?
>>>
>>> *ETS being the obvious exception.
>>>
>>> Secondly, here's a mantra from Joe Armstrong...
>>>
>>> @ minute 17:26
>>> http://www.se-radio.net/2008/03/episode-89-joe-armstrong-on-erlang/
>>>
>>> "[message passing concurrency]... the original reasons have to do with
>>> fault
>>> tolerance... you have to copy all the data you need from computer 1 to
>>> computer 2... if computer 1 crashes you take over on computer 2... you
>>> can't
>>> have dangling pointers... that's the reason for copying everything...
>>> it's
>>> got nothing to do with concurrency, it's got a lot to do with
>>> fault-tolerance... if they don't crash you could just have a dangling
>>> pointer and copy less data but it won't work in the presence of
>>> errors..."
>>>
>>> I interpret this to mean that share-nothing between processes is more
>>> about
>>> replicating valid state than isolating corrupted state as you described.
>>>
>>> Indeed, Joe created an example on his blog...
>>>
>>>
>>> http://armstrongonsoftware.blogspot.com/2007/07/scalable-fault-tolerant-upgradable.html
>>>
>>> It's algorithm 3 there I'm struggling with. Particularly where he says...
>>>
>>> "... In practise we would send an asynchronous stream of messages from N
>>> to
>>> N+1 containing enough information to recover if things go wrong."
>>>
>>> Unfortunately, I couldn't find part II to that post (I don't think there
>>> is
>>> one.) And I'm too green and inexperienced in the field of fault-tolerant
>>> systems to figure it out on my own. I'm having trouble visualising the
>>> practical here from the conceptual -- I need to be shown how :(
>>>
>>> Also, I seem to be under the impression that the Erlang language has some
>>> sort of schematics to do this built-in (i.e. deal with one process taking
>>> over from another if the first fails) and this is the reason processes
>>> share
>>> nothing. This seems to me to be something different from supervision
>>> trees,
>>> which use exit-trapping to re-spawn if a process fails with the active
>>> 'job'
>>> disappearing and any errors logged (like restarting a daemon). My
>>> interpretation of the fault-tolerance Erlang is supposed to enable (for
>>> those in the know) is seamless take-over. The 'job' lives on but
>>> elsewhere.
>>>
>>> Using telecoms as an example: a phone call wouldn't be cut-off when a
>>> fault
>>> occurs, another node would seamlessly take over. This is how I
>>> interpreted
>>> Joe's post and other descriptions of Erlang's fault-tolerant features and
>>> I
>>> understand the key is in the share nothing policy for processes. I'm sure
>>> I've mis-understood something or everything :)
>>>
>>> - Edmond -
>>>
>>> PS: I've read the Manning draft. Great book. I don't know if the answer
>>> lies
>>> in OTP (I searched and didn't find it). I suspect it's lower -- probalby
>>> how
>>> you organise your processes. Some distributed-programming black-magic
>>> only
>>> Erlanger's know about :)
>>>
>>>
>>> On Mon, 03 Jan 2011 14:56:51 +1100, Alain O'Dea <alain.odea@REDACTED>
>>> wrote:
>>>
>>>> On 2011-01-02, at 22:36, "Edmond Begumisa" <ebegumisa@REDACTED>
>>>> wrote:
>>>>
>>>>> Slight correction...
>>>>>
>>>>> On Mon, 03 Jan 2011 12:38:38 +1100, Edmond Begumisa
>>>>> <ebegumisa@REDACTED> wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> I've been trying to wrap my Erlang's fault tolerant features
>>>>>> particularly in relation to processes.
>>>>>>
>>>>>
>>>>> Should be: I've been trying to wrap my head around Erlang's fault
>>>>> tolerant features particularly in relation to processes.
>>>>>
>>>>> Sorry.
>>>>>
>>>>>> I've heard/read repeatedly that the primary reason why Erlang's
>>>>>> designers opted for a share-nothing policy is not rooted in
>>>>>> concurrency but
>>>>>> rather in fault-tolerance. When nothing is shared, everything is
>>>>>> copied.
>>>>>> When everything is copied processes can take over from one another
>>>>>> when
>>>>>> things fail. I follow this reasoning but I don't follow how to apply
>>>>>> it.
>>>>>>
>>>>>> I fully understand and appreciate how supervision trees are used to
>>>>>> restart processes if they fail. What I don't get is what to do when
>>>>>> you
>>>>>> don't want to restart but want to take over, say on another node. I
>>>>>> know
>>>>>> that at a higher-level, OTP has some take-over/fail-over schematics
>>>>>> (at the
>>>>>> application level.) I'm trying to understand things at the processes
>>>>>> level -
>>>>>> why Erlang is the way it is so I can better use it to make my
>>>>>> currently
>>>>>> fault-intolerant program fault tolerant.
>>>>>>
>>>>>> Specifically, how can one process take over from another if it fails?
>>>>>> It
>>>>>> appears to may that the only way to do this would be to somehow
>>>>>> retrieve not
>>>>>> only the state of the process (say, gen_server's state) but also the
>>>>>> messages in its mailbox. Where does the design decision to
>>>>>> share-nothing for
>>>>>> the sake of fault-tolerance come into play for processes? Please help
>>>>>> me
>>>>>> "get" this!
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> - Edmond -
>>>>>>
>>>>>>
>>>>
>>>> Hi Edmond:
>>>>
>>>> Share-nothing helps with concurrent fault-tolerance by preventing one
>>>> process from corrupting the state of another. Receive is a process'
>>>> choice
>>>> and it corrupts its own state if it receives bad data and lets it in.
>>>>
>>>> AFAIK OTP fault-tolerance doesn't mean no requests will fail, it means
>>>> the
>>>> system/sub-system will recover if a single request causes a process to
>>>> crash.  It's kind of like proper try/catch recovery applied to
>>>> concurrent
>>>> code.  How you recover from the crash depends on the supervision
>>>> strategy
>>>> chosen.  In some cases the supervisor can pass the state to the
>>>> replacement
>>>> process. In others this isn't necessary or even desirable since the
>>>> state
>>>> itself may involve resources lost in the crash or corrupted state that
>>>> led
>>>> to the crash.
>>>>
>>>> I am straying outside my knowledge here so this paragraph is guesswork.
>>>>  The message queue for a gen_server need not necessarily be lost when
>>>> the
>>>> callback module crashes.  In theory OTP could (and might already) simply
>>>> delegate the messages to the replacement process following a crash.
>>>>  Someone
>>>> who knows OTP better than me would need to weigh in here though.
>>>>
>>>> I found http://manning.com/logan very informative in understanding OTP
>>>> and
>>>> its supervisor hierarchies.
>>>>
>>>> Cheers,
>>>> Alain
>>>> ________________________________________________________________
>>>> erlang-questions (at) erlang.org mailing list.
>>>> See http://www.erlang.org/faq.html
>>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>>
>>>
>>>
>>> --
>>> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>>>
>>> ________________________________________________________________
>>> erlang-questions (at) erlang.org mailing list.
>>> See http://www.erlang.org/faq.html
>>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>>
>>>
>>
>> ________________________________________________________________
>> erlang-questions (at) erlang.org mailing list.
>> See http://www.erlang.org/faq.html
>> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>>
>
>
> --
> Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
>