[erlang-questions] Investigate an infinite loop on production servers

[Vance Shipley]

On Thu, May 23, 2013 at 04:00:07AM +0200, Morgan Segalis wrote:
}  I'm having a bit of an issue with my production servers.

You will find that etop is your friend:

	http://www.erlang.org/doc/apps/observer/etop_ug.html

Run etop from the command line and sort on the column you're
interested in.  To watch memory usage:

	etop -node tiger@REDACTED -sort memory

This will list the processes by memory size in decreasing order.
This shows you the memory hogs.  Watch it as it starts to get 
into trouble and you should see where the memory is getting used.

As Bob points out the most common problem is that a process's 
inbox will start to fill up.  Once this starts happening it's
the beginning of the end.  Another process may start eating up
memory and the node may crash because it has requested more than
is available bt the root cause was that one process not having
time to service the messages at the rate they are received.

To watch for message queue lengths:

	etop -node tiger@REDACTED -sort msg_q

The above will list the processes in decreasing order of inbox
size.  They should all be zero, and sometimes one, normally.  If
you have a problem you'll see one process stay at the top and it's
message queue length will start to grow over time.

-- 
	-Vance