[erlang-questions] mnesia memory isn't freed after search ?

[Leon de Rooij]

Hi all,

I'm doing some tests with Mnesia, trying to load a milion records and
read them, but apparently memory isn't freed after I do a search. After
a few searches my whole erl shell crashes :(

At the end of this mail is the module I wrote with a test that breaks
when run multiple times and attached is an image of graphs from the
system monitor of Ubuntu.

I started erl (R13B) with:

+A 128
-smp enable
-mnesia dump_log_write_threshold 50000
-mnesia dc_dump_limit 40

I did the following things:
(every peak in the image of CPU history is a search)

1> breakage:init().              % at ~300 sec
{atomic,ok}
2> breakage:fill_table(1000000)  % from ~250 to ~225 sec
..some warnings about mnesia overload..
ok
3> breakage:list_users().        % from ~190 to ~175 sec
ok
4> breakage:list_users().        % from ~165 to ~155 sec
ok
6> breakage:list_users().        % from ~145 to ~130 sec
ok
7> breakage:list_users().        % from ~120 to ~105 sec
ok
8> breakage:list_users().        % from ~95 to ~85 sec
ok
9> breakage:list_users().        % from ~80 to ~70 sec
ok
10> breakage:list_users().       % from ~65 to ~50 sec
ok
11> breakage:list_users().       % from ~40 to ~30 sec
ok
12> breakage:list_users().       % from ~25 to ~20 sec

Crash dump was written to: erl_crash.dump
eheap_alloc: Cannot allocate 3563526520 bytes of memory (of type
"heap").
Aborted

=======================


I don't understand why memory usage isn't the same after as before
list_users() is called.. Actually, the third time when list_users() is
called, the memory drops a bit and then goes back to the level of before
the first search! Is Erlang caching variables even though they went out
of scope ?

I understand it's not wise to read so many records at once, but better
do it in small batches. But still, I assume that if memory isn't freed
after a large search, it won't be after a small one either, it's just
less noticable then, right?

I tried running fill_table(N) with smaller values and then let
quick_break() run for half an hour or so, but it didn't crash then -
also didn't see a noticable memory increase. (Up to 800k goes alright,
850k and up breaks)

Can anyone tell me what I'm doing wrong ?

thanks & kind regards,

Leon de Rooij


=======================

-module(breakage).
-compile(export_all).

-record(user, {username, attrs=[], params=[], vars=[], groups=[]}).

init() ->
        mnesia:create_schema([node()]),
        mnesia:start(),
        mnesia:create_table(user, [
                {disc_copies, [node()]},
                {attributes, record_info(fields, user)}]).

fill_table(N) ->
        F = fun(C) ->
                mnesia:dirty_write(
                        #user{
                                username=erlang:integer_to_list(C),
                                attrs=[{"ak1", "av1"}, {"ak2", "av2"}],
                                params=[{"pk1", "pv1"}, {"pk2", "pv2"}],
                                vars=[{"vk1", "vv1"}, {"vk2", "vv2"}],
                                groups=["group1", "group2"]
                        }
                )
        end,
        lists:foreach(fun(Z) -> F(Z) end, lists:seq(1,N)).

list_users() ->
        mnesia:dirty_match_object({user, '_', '_', '_', '_', '_'}),
        ok.

quick_break() ->
	quick_break(1).
quick_break(I) ->
	io:format("iteration ~p~n", [I]),
	list_users(),
	quick_break(I+1).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot-System Monitor-1.png
Type: image/png
Size: 13892 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090524/83e1783c/attachment.png>