How to get configuration data to a large number of threads?

[Ulf Wiger (AL/EAB)]

An example of Joe's suggestion of moving the functions to
the data can be found in the gen_leader contribution
(at sourceforge). An example program using gen_leader is
gdict -- a fully replicated version of dict. It does 
exactly what Joe proposes: broadcasts a fun which performs
the same operation on all copies of the replicated dictionary.

/Uffe

> -----Original Message-----
> From: owner-erlang-questions@REDACTED
> [mailto:owner-erlang-questions@REDACTED]On Behalf Of Joe Armstrong
> Sent: den 27 oktober 2004 11:33
> To: Heinrich Venter
> Cc: erlang-questions@REDACTED
> Subject: Re: How to get configuration data to a large number 
> of threads?
> 
> 
> 
>    How about some lateral thinking here:
> 
> > I have a transaction based system that spawns a
> > thread to handle every incoming transaction.  Unfortunately there is
> > quite a large chunk of relatively static configuration 
> information that
> > is needed by every thread....
> > The question is, how do I get this information to every 
> thread without
> > significantly slowing things down or using up all the 
> available memory?
> 
> 
>    You don't - you move the functions to the data - not the 
> data to the
> functions :-)
> 
>    Assume the  configuration data is HUGE  (GBytes) - if  
> this case I'd
> send the computations to  the configuration data. Exactly the opposite
> of what you did :-)
> 
>    So you make a global server like this
> 
>    -module(config).
> 
>    start() ->
>  	AHugeDataStructure = read_initiaal_config_data(),
>  	loop(AHugeDataStructure).
> 
> 
>    loop(AHugeDataStructure) ->
>  	receive
>  	   {upgrade, AnotherHugeDataStructure} ->
>  		loop(AnotherHugeDataStruce);
>  	   {compute, From, Fun} ->
>  		Val = (catch Fun(AHugeDataStructure)),
>  	        From ! {self(), Val},
>  	        loop(AHugeDataStructure)
>  	end.
> 
> ... now about the clients ...
> 
>     Suppose the client wants a username and password
> 
>     Make a query
> 
>  	Query = fun(Config) ->
>  		   {get_from_config(username, Config),
>  		    get_from_config(password, Config)
>  		end,
> 
>    Send it to the config process with a
> 
>  	Config ! {compute, self(), Query}
> 
>  	and wait for the reply
> 
> 
>    Note that sending Funs in messages (locally) is a very lightweight
> operation (as I have explained before :-)
> 
>    Now you possibly have a bottleneck in the server - how to 
> solve this?
> 
>    Make several servers with identical copies of the 
> configuration data.
> 
>    Remember - it might be cheaper to move the functions to the data
> than moving the data to the functions.
> 
>    RPC style programming in "Other Languages" (TM) makes you believe
> that there is only only way of doing RPCs (ie moving the data 
> to the functions)
> 
>    Erlang offers freedom of choice.
> 
>    Cheers
> 
> /Joe
> 
> 
> 
> 
> 
>