[erlang-questions] Memory Usage w/disc_copies

[Kevin A. Smith]

I know this has been discussed on the list before but I haven't found  
a satisfactory answer either in the list archives or Google.

I've got some test code which queries 9 disc_copies tables in a tight  
loop. The tables have been preloaded and no new data is created during  
the test. Querying is done via dirty_match_object/1.

I've noticed that the memory footprint of the Erlang VM grows as long  
as I leave my test running. The VM will start with a low memory  
footprint, say around 3-4 MB, and then grow to over 360 MB in the  
space of a minute. I know I could use ets or dets directly and  
probably workaround this issue but I'm intrigued and I'd like to  
understand exactly what's happening.

erlang:memory/0 reports this after a typical test run:

[{total,385341784},
  {processes,694596},
  {processes_used,686596},
  {system,384647188},
  {atom,402833},
  {atom_used,372656},
  {binary,13728},
  {code,2993679},
  {ets,380521792}]

It looks like ets is responsible for the lion's share of memory usage.

I've tried all of the standard tricks:

* setting fullsweep_after via erlang:system_flag/2 (before any test  
code is run)
* calling erlang:garbage_collect/0 at the top of each loop iteration
* calling mnesia_recover:allow_garb/0 and mnesia_recover:start_garb/0  
frequently

None of these changes have resulted in any measurable improvement. I  
did find this post from last year:

http://tinyurl.com/mnesia

I've verified that Dan Gudmundsson's changes (see the end of the above  
thread) are present in mnesia_dumper and mnesia_recover. I'm running  
R12B-5 so it looks like Dan's changes were incorporated at some point.

The odd thing is if I change the table type to disc_only_copies the  
problem goes away. Memory usage peaks at around 20 MB and averages out  
around 11 MB.

Can anyone shed some light on the behavior I'm seeing?

Thanks,
Kevin