[erlang-bugs] R11B-5 to R12B-3 forks beams eating 100% CPU

[Lev Walkin]

SYMPTOMS:

	A running, unstressed erlang system appears to spontaneously
	fork off BEAM processes (beam) several times an hour,
	each eating 100% CPU. (If there are more than one at a time,
	they split the CPU accordingly).

INVESTIGATION:

	The BEAM processes forked appear to be truly separate processes.
	Killing them (-9) does not seem to harm the main VM, which
	continues to execute the necessary set of applications
	(kernel, stdlib, sasl, ssl, yaws, plus two inhouse ones).

	Disabling yaws application does not seem to have any effect on
	the rate of runaway beam creation.

	ktrace shows a total absence of any system calls in the runaway
	beam process, suggesting an infinite loop. Killing the process
	results in a single ktrace event:

	[root@REDACTED ~]# kdump
	 97445 beam     PSIG  SIGKILL SIG_DFL
	[root@REDACTED ~]#

	The R12B-3 has the following fix:


	    OTP-7289  On Mac OS 10.5 (Leopard), sending to socket
	              which the other end closes could cause the
	              emulator to consume 100% CPU time.
	              (Thanks to Matthias Radestock.)

	Since Mac OS is sufficiently similar to FreeBSD, I had hoped
	the OTP-7289 has direct relation to my runaway beam problem.
	Alas, R12B-3 behaves exactly the way R11B-5 and the latter
	versions of Erlang VM behave, that is, runaway beams
	are continuing to appear on a regular basis.

WORKAROUND USED:

	Created an external process which periodically detects
	and kills the runaway beams. Ugly.

CONFIGURATION:

	Erlang (BEAM) emulator version 5.6.3 [source] [64-bit] 
[async-threads:0] [hipe] [kernel-poll:false]

	FreeBSD host 6.3-RELEASE FreeBSD 6.3-RELEASE #0: Wed Jan 16 01:43:02 
UTC 2008     root@REDACTED:/usr/obj/usr/src/sys/SMP  amd64

	However, the system was running all versions of erlang VM
	between R11B-5 and R12B-3 with the same results, so
	it is not version specific.


Please suggest further steps to debug or eliminate the problem.


-- 
Lev Walkin
vlm@REDACTED