[erlang-questions] Memory Usage w/disc_copies

[Kevin A. Smith]

Right. I knew that disc_copies were RAM resident so I'd expect some  
increase in memory usage. The behavior I'm seeing is that successive  
identical queries which return the same results each time consume more  
memory with each invocation. If the data is in a steady state I'd  
expect the memory footprint to level off at some point, too. That's  
not happening and I don't understand why.

--Kevin
On Feb 19, 2009, at 5:09 AM, Scott Zhang wrote:

> Hi. Kevin.
> I am not sure your exact question is. But based on my experience  
> with mnesia with table type disc_copies. mnesia will keep a complete  
> copy of table in ram too. The first time you access that table, 
> (Ususally I am using tv:start()). It reside in RAM. Comsume your RAM  
> until it crash your erlang VM.
>
> It is a potential dangerous to use ram_copies and disc_copies,  
> because unconscious it will crash itself.
>
>
> Regards.
> Scott
>
>
>
> On Thu, Feb 19, 2009 at 12:38 PM, Kevin A. Smith <kevin@REDACTED 
> > wrote:
> I know this has been discussed on the list before but I haven't found
> a satisfactory answer either in the list archives or Google.
>
> I've got some test code which queries 9 disc_copies tables in a tight
> loop. The tables have been preloaded and no new data is created during
> the test. Querying is done via dirty_match_object/1.
>
> I've noticed that the memory footprint of the Erlang VM grows as long
> as I leave my test running. The VM will start with a low memory
> footprint, say around 3-4 MB, and then grow to over 360 MB in the
> space of a minute. I know I could use ets or dets directly and
> probably workaround this issue but I'm intrigued and I'd like to
> understand exactly what's happening.
>
> erlang:memory/0 reports this after a typical test run:
>
> [{total,385341784},
>  {processes,694596},
>  {processes_used,686596},
>  {system,384647188},
>  {atom,402833},
>  {atom_used,372656},
>  {binary,13728},
>  {code,2993679},
>  {ets,380521792}]
>
> It looks like ets is responsible for the lion's share of memory usage.
>
> I've tried all of the standard tricks:
>
> * setting fullsweep_after via erlang:system_flag/2 (before any test
> code is run)
> * calling erlang:garbage_collect/0 at the top of each loop iteration
> * calling mnesia_recover:allow_garb/0 and mnesia_recover:start_garb/0
> frequently
>
> None of these changes have resulted in any measurable improvement. I
> did find this post from last year:
>
> http://tinyurl.com/mnesia
>
> I've verified that Dan Gudmundsson's changes (see the end of the above
> thread) are present in mnesia_dumper and mnesia_recover. I'm running
> R12B-5 so it looks like Dan's changes were incorporated at some point.
>
> The odd thing is if I change the table type to disc_only_copies the
> problem goes away. Memory usage peaks at around 20 MB and averages out
> around 11 MB.
>
> Can anyone shed some light on the behavior I'm seeing?
>
> Thanks,
> Kevin
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>