Question relating to trace/debug/log OTP features

[Reto Kramer]

I'm puzzled by the many facilities OTP contains to generate trace-, 
debug- and log information. Perhaps someone could point me to an 
overview/tutorial of what the mechanisms are that you recommend I use 
for a brand new development.  What's missing the most in the docs (or I 
could not find it) is how the various mechanisms relate to each other 
(e.g. which ones are basic services that high level trace/log services 
leverage, which ones are "old" and still there for backwards 
compatibility).

Specifically I need 3 things:
1. error/warning/info/detail messages that the end user should see in 
production.
2. debug information.
3. more debug information (free form, not tied to gen_* 
entry/exit/state-change)

For (1) I use error_logger currently, is there an alternative I should 
consider?  Sometimes I'd like to be able to show customers more levels 
of granularity (currently error and info are available, I need warning 
and detail as well) without having to resort to the debug level data in 
(2) and (3).  Is there a way to add another level to the current 2 
(error/info/...)?  Is there an Erlang/OTP philosophy to only having 2 
levels?
For (2)  I use sys:trace(Mod,true) to enable *DBG* messages for gen_* 
behaviour and for my own (along section 6.2.1 "Special processes" of 
the "design principles" doc). However sometimes I'd like to show 
additional *DBG* messages of my own in callback modules that I pass to 
gen_* behaviour - how can I do that (i.e. what do I call inside the 
callback module to generate a *DBG* event?
For (3) I currently use a macro to enable/disable io:format calls (the 
old style "recompile with debug" story). Is there a facility in OTP 
that I can use to enable dumping of debug information at runtime w/o 
recompiling (I'm quite happy to accept the performance penalty, my 
application is not performance critical)? E.g. can  I use sys:trace in 
arbitrary code and through that get the dynamic enable/disable?

In addition, does anyone have examples code or tutorial on the 
following (or advice as to when to use):
- disk_log and disk_log_wrapper (i.e. vs. the error_logger info/error 
messages)
- seq_trace and the observer/ttb (R9C). Specifically are seq_trace and 
ttb the basis and observer the visualizer? How do I invoke the observer 
(R9C docs are silent on that)?
- How does et (et_collector, selector, viewer) relate to the 
observer/ttb/seq_trace facilities?  How does it relate to sys:trace?

I'm hoping that some OTP expert can help me understand these facilities 
and their relation and when to use what (ideally by 
example/experience).  Perhaps someone could talk about the OTP history 
and how the many powerful mechanisms were motivated (various projects, 
timing, ...)

Thanks in advance for helping me navigate the Zoo.
- Reto
_________________
A distributed system is a system in which I can't do my work because 
some computer has failed that I've never even heared of . -- Leslie 
Lamport
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/enriched
Size: 3064 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20031209/dae3094c/attachment.bin>