geometric memory growth

[Ulf Wiger]

(My third try sending this mail to the list.
Apologies if you get it more than once, but after two days, I have to
assume that the first post got lost somewhere.)


Here's a subtle bug that caused me about one day's worth of headache:

I have process that holds a gen_tcp session, and multiplexes "virtual
sessions" on top of it.

When running a benchmark, I noticed exponential memory growth, and it took
me a while to notice that the growth was actually taking place in the
innocent benchmark loop (about 7 lines of code, basically this:

vcc_rpc2(Channel) ->
    vccTransport:msg(Channel, 0, vcc_rex, {self(), ping},
		     [{temp_channel,true}]),
    receive
	{_, _, pong} ->
	    ok;
    end.

which amounts to an ets:lookup, a gen_server:call() and waiting for a
message where the third element is
'pong')

17 iterations made the VM allocate 1 GB of RAM.

The problem? (finally diagnosed once Björn G tipped me off about
erts_debug:size(Term) - thanks Björn!)

The second element of the message was a fun:

    SendF = fun(To, Msg) ->
	          msg(State#state.channel,
	          ChNo, To, Msg)
	      end,

Horrid, isn't it?

Writing it this way, causes the _whole_ State to be bound inside the fun.
And State of course contained, among other things, the metadata for the
virtual sessions (e.g. other SendF instances.)

This wasn't much of a problem as long as everything was local references
on the same heap (it probably would have shown after a while, though), but
when the fun was passed in a message to another process, all relative
references were flattened out, and the size of the fun doubled in each
iteration.


The obvious fix:

    Channel = State#state.channel,
    SendF = fun(To, Msg) ->
	          msg(Channel,
	          ChNo, To, Msg)
	      end,

Now, wouldn't it be great if the compiler could figure out how to do this
at compile-time?

/Uffe