How to get configuration data to a large number of threads?

[Raimo Niskanen]

Just a quick suggestion, has probably been suggested before.

Store configuration data in an Ets table. Wrap it with some slik API.
Request configuration data in reasonable bits when needed. These bits
will get copied but if you do not use all configuration data all the
time - when a process needs more heap it will first GC which will
free unreferenced configuration data and hopefully the heap will not
need to grow.

This will probably happen if you use the OTP framework with 
application:get_env() that I heard store data in Ets tables.
But if you have own wrappers to Ets you may organize configuration
data so you can read just what is needed to decide what is 
needed next. Depends on your problem.



heinrich@REDACTED (Heinrich Venter) writes:

> A bit more detail on the problem is in order I see :)
> 
> Each incoming transaction spwawns a process that handles it up to a
> point where output is generated.  We need to be able to handle large
> volumes of transactions in a short time in bursts (SMS system using
> oserl http://oserl.sourceforge.net/)
> Each process needs to interpret the transaction content based on a set
> of configuration data.  Unfortunately the entrie config set is needed
> for this and it could be as large as 5k (worst case, 1-2k is probably
> more realistic.)
> Some quick profiling shows that we can expect in the order of thousands
> of processes active at the same time, making the memory overhead a
> problem.
> In order to keep the processing speed as fast as possible I want as many
> prallel processes as I can manage.  Obviosly if I can get clever with
> the config data, this would mean more processes.  Failing that, I will
> have to place a lower limit on the number of processes so that they will
> fit into memory or to prevent memory churn.
> 
> From the coments up to now it seems that I am most likely going to have
> to be satisfied with passing the config data as a parameter to the
> function that executes in each thread and rely on the GC to keep things
> sane.
> 
> -]-[einrich
> 
> 
> "Vodacom Look4me - Click here for more info"
> http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313533
> 
> Bought to you by Launchpad.
> http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313438

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB