FW: How to get configuration data to a large number of threads?

[Ulf Wiger (AL/EAB)]

Apologies. I meant to send this to the list only; instead I sent
it to Heinrich only...
 
Apologies, Heinrich, for sending you two copies of the same msg.
 
/Uffe
 
-----Original Message-----
From: Ulf Wiger (AL/EAB) 
Sent: den 26 oktober 2004 17:41
To: 'Heinrich Venter'
Subject: RE: How to get configuration data to a large number of threads?


 
You could possibly follow Joe's advice and play around with the 
command line options 'erl -shared' and 'erl -hybrid' in OTP R10B.
 
The '-shared' option will activate a "shared-heap" implementation,
where all processes operate on the same memory heap. For some
applications, this can significantly increase performance and/or 
reduce memory consumption. Notably, message passing cost is
always constant, since one only needs to pass a pointer to a 
shared data structure. A downside to "shared heap" is that the 
garbage collector has not been optimized for it (an incremental
garbage collector would probably be the best choice.) Another
downside is of course that it's an experimental feature, i.e. not 
supported.
 
The '-hybrid' option keeps a separate heap for each process, but
attempts to put data structures used in message passing on a 
shared heap. This should give some of the advantages with 
"shared heap" but reduce the garbage collection problems.
 
NOTE: when I tried the two options on my own OTP R10,
the '-hybrid' option failed. It seems that 'make' successfully
built beam.hybrid, but 'make install' didn't copy it to the 
official erts-5.4/bin/ directory. Easy to fix. Forgive me for not
submitting a patch.  (-:
 
Regards,
Uffe

-----Original Message-----
From: owner-erlang-questions@REDACTED [mailto:owner-erlang-questions@REDACTED]On Behalf Of Heinrich Venter
Sent: den 26 oktober 2004 16:32
To: erlang-questions@REDACTED
Subject: RE: How to get configuration data to a large number of threads?


 <http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313533> 
 <http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313438> Bought to you by Launchpad. 

A bit more detail on the problem is in order I see :)

Each incoming transaction spwawns a process that handles it up to a
point where output is generated. We need to be able to handle large
volumes of transactions in a short time in bursts (SMS system using
oserl http://oserl.sourceforge.net/)
Each process needs to interpret the transaction content based on a set
of configuration data. Unfortunately the entrie config set is needed
for this and it could be as large as 5k (worst case, 1-2k is probably
more realistic.)
Some quick profiling shows that we can expect in the order of thousands
of processes active at the same time, making the memory overhead a
problem.
In order to keep the processing speed as fast as possible I want as many
prallel processes as I can manage. Obviosly if I can get clever with
the config data, this would mean more processes. Failing that, I will
have to place a lower limit on the number of processes so that they will
fit into memory or to prevent memory churn.

>From the coments up to now it seems that I am most likely going to have
to be satisfied with passing the config data as a parameter to the
function that executes in each thread and rely on the GC to keep things
sane.

-]-[einrich 

 <http://www.rocketseed.com/discstart.html> 




 <http://www.rocketseed.com/discend.html> 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/19118cda/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MTk2LjMwLjc5LjE1NQ==
Type: image/gif
Size: 16471 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/19118cda/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MTk2LjMwLjc5LjE1NQ==
Type: image/gif
Size: 557 bytes
Desc: not available
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20041026/19118cda/attachment-0001.gif>