Memory usage with ETS

[Peter-Henry Mander]

Hi Uffe,

On Mon, 2005-08-08 at 09:17 +0200, Ulf Wiger wrote:

> ETS tables dynamically expand, but they should also shrink
> dynamically (with a hysteresis.) Are you sure you don't have
> a memory leak in the sense that you have lingering objects
> that should have been deleted?

It's not a bug in ETS, it's a flaw in my program. I found that I was
using 'bag' instead of 'set' and the memory leak was due to multiple
record instances not being retrieved and deleted properly. Like you
said, I had 'lingering objects'. Why did I choose 'bag'? I wonder...

When I used the 'set' table option the memory consuption was much
improved, it didn't monotonically increase over time.

> Hibernate in a process holding only a fun should be quite close to the
> memory footprint of the same fun in an ETS table. Note that if you
> create the process with a proc_lib function, you will get some stuff
> in the process dictionary that, while being useful for debugging,
> of course will increase the footprint. Hibernate doesn't empty the
> process dictionary.

I'm a good lad, I avoid using the process dictionary altogether. How can
I ensure that the process context only contains the strict minimum
before hibernating? Is there a way of purging the process dictionary?

> Regarding the number of messages, using spawn/hibernate is going to
> use less CPU power than the timer.erl module. That should be the
> important part (that, and memory, of course).

Alright, I'll try to write up a comparative test of the two methods, one
using my micro_timer, and the other using timer BIFs and hibernate.

The micro_timer works quite well, except I've observed some spurious
delays over long time periods. Would garbage collection potentially
cause these delays?

I realise that insisting on implementing this in Erlang may be
misguided. An old 'C' dog would turn his nose up at the idea of
implementing time-critical software in a declarative language with
periodic garbage collection. But I've had encouraging results so far,
and I wish to iron out the remaining wrinkle.

Pete.