How to manage 100000+ Megaco contexts on a measly PC?

[Peter-Henry Mander]

Hi Gurus,

This is a simple question, hopefully with a simple answer that I
probably would have concluded for myself. But it's Friday, yesterday was
a company celebration, and I'm lazy. So apologies all around if I sound
brain-dead.

I'm using the OTP/Megaco stack in Erlang to perform functional and
stress test of our Session Controller. I wish to create over 100000
contexts with two terminations each. So far, due to memory constraints
alone, I can only reach 5000 contexts before virtual memory kicks in and
thrashes the disk destroying the otherwise acceptable rate of 300~400
context creations each second for each PC. We can already use multiple
PCs to exceed the rate of the Session Controller, but we don't achieve
the 100000 or more concurrent contexts which would require all the PCs
we currently have to test just one machine!

So here are the strategies I've devised so far:

1) A very simple solution would be to stuff 16GB or more RAM into the
PC. It's cheap but not elegant, and I object to brute force solutions
where better algorithms would pay much greater return.

2) Another solution is to reduce the memory footprint of each process
mirroring each context in the Session Controller. How would I do this?
Would a garbage-collection once the context is created (Megaco Add) and
configured (Megaco Modify) reduce the memory footprint sufficiently to
allow 100000 context processes to co-exist in 1GB of RAM?

3) The solution I think would be far better than either the above would
be to store the Context ID in a list or database after completing the
Modify and squirting a burst of RTP media through the Session Controller
to verify proper operation, then terminate the process and delegate to a
subtraction process which would recover the Context ID at the
appropriate moment and issue a Subtract to the Session Controller. All
that is required for Subtract is the Context ID, which need to be sorted
and stored according to subtraction order. Since the subtract rate is
the same as the add rate I can use a second instance of the rate-control
process that I already wrote. Since I need a FIFO to store context IDs,
is there one available in OTP? I would prefer to avoid continually
reversing a list of over 100000 elements!

Has anyone else confronted a similar issue and arrived at the same
conclusion that (3) is the best solution, or is there an even better or
simpler solution?

Pete.

-- 
"The Tao of Programming
 flows far away 
 and returns 
 on the wind of morning."