<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>Untitled</TITLE>
<META content="MSHTML 6.00.2800.1458" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=594294115-26102004>Apologies. I meant to send this to the list only;
instead I sent</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=594294115-26102004>it to
Heinrich only...</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=594294115-26102004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=594294115-26102004>Apologies, Heinrich, for sending you two copies of the
same msg.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=594294115-26102004></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=594294115-26102004>/Uffe</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B> Ulf Wiger (AL/EAB)
<BR><B>Sent:</B> den 26 oktober 2004 17:41<BR><B>To:</B> 'Heinrich
Venter'<BR><B>Subject:</B> RE: How to get configuration data to a large number
of threads?<BR><BR></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>You
could possibly follow Joe's advice and play around with the </FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>command line options 'erl -shared' and 'erl -hybrid' in OTP
R10B.</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>The
'-shared' option will activate a "shared-heap"
implementation,</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>where
all processes operate on the same memory heap. For some</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>applications, this can significantly increase performance and/or
</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>reduce
memory consumption. Notably, message passing cost is</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>always
constant, since one only needs to pass a pointer to a </FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>shared
data structure. A downside to "shared heap" is that the </FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>garbage collector has not been optimized for it (an
incremental</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>garbage collector would probably be the best choice.)
Another</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>downside is of course that it's an experimental feature, i.e. not
</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>supported.</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>The
'-hybrid' option keeps a separate heap for each process, but</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>attempts to put data structures used in message passing on a
</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>shared
heap. This should give some of the advantages with </FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>"shared heap" but reduce the garbage collection
problems.</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>NOTE:
when I tried the two options on my own OTP R10,</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>the
'-hybrid' option failed. It seems that 'make' successfully</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff size=2>built
beam.hybrid, but 'make install' didn't copy it to the </FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>official erts-5.4/bin/ directory. Easy to fix. Forgive me for
not</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>submitting a patch. (-:</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>Regards,</FONT></SPAN></DIV>
<DIV><SPAN class=115322615-26102004><FONT face=Arial color=#0000ff
size=2>Uffe</FONT></SPAN></DIV>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B>
owner-erlang-questions@erlang.org
[mailto:owner-erlang-questions@erlang.org]<B>On Behalf Of </B>Heinrich
Venter<BR><B>Sent:</B> den 26 oktober 2004 16:32<BR><B>To:</B>
erlang-questions@erlang.org<BR><B>Subject:</B> RE: How to get configuration
data to a large number of threads?<BR><BR></FONT></DIV><A
href="http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313533"><!-- RS-STRIP-BEGIN MTk2LjMwLjc5LjE1NQ== --><IMG
alt="" src="cid:115322615@26102004-1e40" border=0 Vodacom Look4me - Click here
for more info??><!-- RS-STRIP-END MTk2LjMwLjc5LjE1NQ== --></A><BR><A
href="http://asp1.rocketseed.com/RocketSeed/mail/433a32353a3733373030313a33303639373a2d323a313438"><!-- RS-STRIP-BEGIN MTk2LjMwLjc5LjE1NQ== --><IMG
alt="Bought to you by Launchpad." src="cid:115322615@26102004-1e47" border=0><!-- RS-STRIP-END MTk2LjMwLjc5LjE1NQ== --></A>
<P>A bit more detail on the problem is in order I see :)<BR><BR>Each incoming
transaction spwawns a process that handles it up to a<BR>point where output is
generated. We need to be able to handle large<BR>volumes of transactions in a
short time in bursts (SMS system using<BR>oserl
http://oserl.sourceforge.net/)<BR>Each process needs to interpret the
transaction content based on a set<BR>of configuration data. Unfortunately the
entrie config set is needed<BR>for this and it could be as large as 5k (worst
case, 1-2k is probably<BR>more realistic.)<BR>Some quick profiling shows that
we can expect in the order of thousands<BR>of processes active at the same
time, making the memory overhead a<BR>problem.<BR>In order to keep the
processing speed as fast as possible I want as many<BR>prallel processes as I
can manage. Obviosly if I can get clever with<BR>the config data, this would
mean more processes. Failing that, I will<BR>have to place a lower limit on
the number of processes so that they will<BR>fit into memory or to prevent
memory churn.<BR><BR>From the coments up to now it seems that I am most likely
going to have<BR>to be satisfied with passing the config data as a parameter
to the<BR>function that executes in each thread and rely on the GC to keep
things<BR>sane.<BR><BR>-]-[einrich </P><A
href="http://www.rocketseed.com/discstart.html"><BR></A>
<P></P><A href="http://www.rocketseed.com/discend.html"><BR></A>
<P></P></BLOCKQUOTE></BODY></HTML>