fault-tolerance

[Joe Armstrong]

On Mon, 29 Mar 2004, Clara Benac Earle wrote:

> Hi,
> 
> I have written a simple client-server system in Erlang using the generic 
> server behaviour. My system should recover gracefully when a client 
> process dies. To do that I am using the supervsion tree behaviour for 
> restarting the client process and the process_flag, link and handle_info 
> primitives in the server to trap the exit signal from the client process 
> and delete the information about the crashed process from the state of 
> the server.
> 
> Now, I am having difficulties to test my program. To test the fault 
> tolerance all I am doing is to start pman and kill a process client and 
> observe the messages I am printing on the screen. The other way I am 
> testing it is by writing code in the client which will make the client 
> die. In this way I have a better knowledge of at which point the client 
> dies. However, none of this two ways of debugging is very satisfactory.
> 
> I wanted to ask what people does for debugging fault-tolerant programs.

Well if your problem is that the messages roll off the screen you can 
need to save them on disk so you can read them later.

Quick hack: If you have a *nix box you can type "script" in a shell window
before you do anything - after that run any commands - type Control D and
everything will be saved in a file called "typescript".

Better: write trace messages a file instead of the terminal

Even better: write trace message to dets so you can analyze them later

Even more better: write trace message to mnesia

Your trace messages should be Erlang terms so that another program can later
analyze them (if the problem is complex)

Then you need to think about *invariants*

Invariants are things that are always true - you must analyze your problem
to identify the things that are always true - and then write code to test
that that the invariants hold.

Finally - *use the coverage analyzer*

  Do erl -man cover

IMHO coverage is the single best technique for making production quality code.
 
Cheers

/Joe

> 
> Cheers,
> Clara
> 
> 
> This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.
> 
> E-mail including attachments is susceptible to data corruption, interruption, unauthorized amendment, tampering and viruses, and we only send and receive e-mails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.
>