geometric memory growth

[Ulf Wiger (AL/EAB)]

(My second try sending this mail to the list.
Apologies if you get it twice, but after two
days, I have to assume that the first post got
lost somewhere.)


Here's a subtle bug that caused me about one day's worth of headache:

I have process that holds a gen_tcp session, and multiplexes "virtual sessions" on top of it.

When running a benchmark, I noticed exponential memory growth, and it took me a while to notice that the growth was actually taking place in the innocent benchmark loop (about 7 lines of code, basically this:

vcc_rpc2(Channel) ->
    vccTransport:msg(Channel, 0, vcc_rex, {self(), ping},
		     [{temp_channel,true}]),
    receive
	{_, _, pong} ->
	    ok;
    end.

which amounts to an ets:lookup, a gen_server:call() and waiting for a message where the third element is
'pong')

17 iterations made the VM allocate 1 GB of RAM.

The problem? (finally diagnosed once Björn G tipped me off about erts_debug:size(Term) - thanks Björn!)

The second element of the message was a fun:

    SendF = fun(To, Msg) ->
	          msg(State#state.channel,
	          ChNo, To, Msg)
	      end,

Horrid, isn't it?

Writing it this way, causes the _whole_ State to be bound inside the fun. And State of course contained, among other things, the metadata for the virtual sessions (e.g. other SendF instances.)

This wasn't much of a problem as long as everything was local references on the same heap (it probably would have shown after a while, though), but when the fun was passed in a message to another process, all relative references were flattened out, and the size of the fun doubled in each iteration.


The obvious fix:

    Channel = State#state.channel,
    SendF = fun(To, Msg) ->
	          msg(Channel,
	          ChNo, To, Msg)
	      end,

Now, wouldn't it be great if the compiler could figure out how to do this at compile-time?

/Uffe