[erlang-questions] Memory Usage

[Ulf Wiger]

Andrew Thompson wrote:
> On Tue, Sep 29, 2009 at 10:11:57AM -0600, Erickson, John wrote:
>> Hi, I am trying to store a large list of items and am a little
>> surprised how much memory erlang is using.  As a test, I tried
>> creating a list of 10M integers and measuring memory usage.  It
>> takes around 600M.  If I convert this list to a binary using
>> term_to_binary, it is only about 50M, which is more like what I
>> would expect.  So I am wondering why there is 10x memory usage in
>> the usual form?  One other small oddity, it seems after calling
>> garbage_collect I am using more memory.
>> 
> 
> Erlang lists incur about an 8x expansion in memory usage since each 
> element is a pair of 32 bit integers.

There is also a general problem of building large terms
on the process heap. Garbage collection will kick in each
time the heap fills up, and it will sweep for garbage to
reclaim. Since there is no garbage, it will fail, and try
a 'full sweep', which also checks long-lived data. Since
that doesn't find any garbage either, the VM will resize
the heap instead. Since the VM cannot foresee that the
heap will continue to grow, it will repeat this pattern
for each GC, ending up making several copies of the heap
each time.

For this reason, one should try to favor algorithms that
execute in roughly constant space - e.g. fetching one
or a few objects at a time from ETS, rather than pulling
the entire set at once. One can also try to size the
heap at spawn-time (using spawn_opt) to avoid GC whenever
possible.

BR,
Ulf W
-- 
Ulf Wiger
CTO, Erlang Training & Consulting Ltd
http://www.erlang-consulting.com