How to manage 100000+ Megaco contexts on a measly PC?

[Bjorn Gustavsson]

Have you looked at erlang:hibernate/3, introduced in R9C?

/Bjorn

Peter-Henry Mander <erlang@REDACTED> writes:

> Hi Gurus,
> 
> This is a simple question, hopefully with a simple answer that I
> probably would have concluded for myself. But it's Friday, yesterday was
> a company celebration, and I'm lazy. So apologies all around if I sound
> brain-dead.
> 
> I'm using the OTP/Megaco stack in Erlang to perform functional and
> stress test of our Session Controller. I wish to create over 100000
> contexts with two terminations each. So far, due to memory constraints
> alone, I can only reach 5000 contexts before virtual memory kicks in and
> thrashes the disk destroying the otherwise acceptable rate of 300~400
> context creations each second for each PC. We can already use multiple
> PCs to exceed the rate of the Session Controller, but we don't achieve
> the 100000 or more concurrent contexts which would require all the PCs
> we currently have to test just one machine!
> 
> So here are the strategies I've devised so far:
> 
> 1) A very simple solution would be to stuff 16GB or more RAM into the
> PC. It's cheap but not elegant, and I object to brute force solutions
> where better algorithms would pay much greater return.
> 
> 2) Another solution is to reduce the memory footprint of each process
> mirroring each context in the Session Controller. How would I do this?
> Would a garbage-collection once the context is created (Megaco Add) and
> configured (Megaco Modify) reduce the memory footprint sufficiently to
> allow 100000 context processes to co-exist in 1GB of RAM?
> 
> 3) The solution I think would be far better than either the above would
> be to store the Context ID in a list or database after completing the
> Modify and squirting a burst of RTP media through the Session Controller
> to verify proper operation, then terminate the process and delegate to a
> subtraction process which would recover the Context ID at the
> appropriate moment and issue a Subtract to the Session Controller. All
> that is required for Subtract is the Context ID, which need to be sorted
> and stored according to subtraction order. Since the subtract rate is
> the same as the add rate I can use a second instance of the rate-control
> process that I already wrote. Since I need a FIFO to store context IDs,
> is there one available in OTP? I would prefer to avoid continually
> reversing a list of over 100000 elements!
> 
> Has anyone else confronted a similar issue and arrived at the same
> conclusion that (3) is the best solution, or is there an even better or
> simpler solution?
> 
> Pete.
> 
> -- 
> "The Tao of Programming
>  flows far away 
>  and returns 
>  on the wind of morning."
> 

-- 
Björn Gustavsson, Erlang/OTP, Ericsson AB