How to get configuration data to a large number of threads?

[Taavi Talvik]

On Tue, 26 Oct 2004, Heinrich Venter wrote:

> A bit more detail on the problem is in order I see :)
>
> Each incoming transaction spwawns a process that handles it up to a
> point where output is generated.  We need to be able to handle large
> volumes of transactions in a short time in bursts (SMS system using
> oserl http://oserl.sourceforge.net/)
> Each process needs to interpret the transaction content based on a set
> of configuration data.  Unfortunately the entrie config set is needed
> for this and it could be as large as 5k (worst case, 1-2k is probably
> more realistic.)
> Some quick profiling shows that we can expect in the order of thousands
> of processes active at the same time, making the memory overhead a
> problem.
> In order to keep the processing speed as fast as possible I want as many
> prallel processes as I can manage.  Obviosly if I can get clever with
> the config data, this would mean more processes.  Failing that, I will
> have to place a lower limit on the number of processes so that they will
> fit into memory or to prevent memory churn.

You have to manage number of parallel processes or load sharing
anyway.

Reasonable approach is to queue incoming messages and forward
them to futher processing according to load (memory footprint,
load balancing in case of multiple prossessing nodes etc).

Then you get graceful degradation in performance.


best regards,
taavi