[erlang-questions] "fat burner" GC on periodic base?

[Valentin Nechayev]

Our application area specifics is combination of high load and rather
complex processing of data.

When investigating a VM, I have found unusually big process memory usage:

2012-04-24T13:42:21.389762+0400 Top 20 consumers by total_heap_size:
2012-04-24T13:42:21.390460+0400 1) pid=<0.29.0> total_heap_size=105953765 desc=user
2012-04-24T13:42:21.390655+0400 2) pid=<0.6.0> total_heap_size=84274435 desc=error_logger
[...]

This confused me and I explicitly called GC on them, and have got after that:

(sysmon7@REDACTED)8> process_info(pid(0,29,0),total_heap_size).
{total_heap_size,3571}
(sysmon7@REDACTED)9> process_info(pid(0,6,0),total_heap_size). 
{total_heap_size,233}

This shrinking by ~1.5G by simple explicit GC call is shocking. To
avoid such situations, I'm now probing of activating periodic GC
process sweep by default (as opposed to previous mode when it needed to
be explicitly enabled per VM).
The burner is as follows:

==={{{
fat_burner() ->
    mmc_lib:mmc_desc({?MODULE, fat_burner}),
    put('$initial_call', {?MODULE, fat_burner, 0}),
    erlang:register(mmc_foreman_fat_burner, self()),
    fat_burner_loop(0, 0).

fat_burner_loop(PrevSpent, PrevLimit) ->
    timer:sleep(?FAT_BURNER_INTERVAL_BASE + PrevSpent),
    Processes = erlang:processes(),
    MemorySum = fb_get_total_memory(Processes),
    if
        MemorySum < PrevLimit ->
            fat_burner_loop(PrevSpent, PrevLimit);
        true ->
            TStart = now(),
            lists:foreach(
                    fun(Pid) -> catch erlang:garbage_collect(Pid) end,
                    Processes),
            TFinish = now(),
            NewSpent = trunc(1000*mmc_lib:now_diff_seconds(TFinish, TStart)),
            NewLimit = 2 * fb_get_total_memory(Processes),
            fat_burner_loop(NewSpent, NewLimit)
    end.

fb_get_total_memory(Processes) ->
    lists:foldl(
        fun(Pid, AccIn) ->
                PM = case catch erlang:process_info(Pid, memory) of
                    {memory, PMX} -> PMX; _ -> 0
                end,
                PM + AccIn
        end, 0, Processes).
===}}}

My questions are:

1. Is there another way to achieve the same result, as avoiding keeping
useless data, on timed basics instead of reduction count, which, as
shown above, doesn't work enough?

2. Whether the algorithm shown in the code snippet should be fixed?
Currently it clones approach of Lua which is directly targeted for
reasonably minimal memory footprint in an environment with Unix styled
VM, combined with calling GC only when memory usage is definitely grown
(2 times - hardcoded here).


-netch-