[erlang-questions] Memory Usage

[Andrew Thompson]

On Tue, Sep 29, 2009 at 10:11:57AM -0600, Erickson, John wrote:
> Hi, I am trying to store a large list of items and am a little surprised how much memory erlang is using.  As a test, I tried creating a list of 10M integers and measuring memory usage.  It takes around 600M.  If I convert this list to a binary using term_to_binary, it is only about 50M, which is more like what I would expect.  So I am wondering why there is 10x memory usage in the usual form?  One other small oddity, it seems after calling garbage_collect I am using more memory.
>

Erlang lists incur about an 8x expansion in memory usage since each
element is a pair of 32 bit integers. I just dealt with this myself in
my gen_smtp erlang email server. I was having a terrible time reading in
large emails (3mb or larger) because of the ballooning memory usage. I
rewrote everything to use binaries wherever possible (had to write some
functions for working with binary strings) but I've seen a huge
reduction in the time and memory needed to handle large emails.

So yeah, binaries are a much better solution for any larger data,
they're just a little tricker to work with because the bitstring library
proposed by EEP9 doesn't seem to exist.

Andrew