How to get configuration data to a large number of threads?

[Inswitch Solutions - Erlang Evaluation]

Hi,

Why is not application:get_env/2 a good solution for huge configuration data
management?


thanks,
Eduardo Figoli
INSwitch Solutions


----- Original Message -----
From: "Joe Armstrong" <joe@REDACTED>
To: "Heinrich Venter" <heinrich@REDACTED>
Cc: <erlang-questions@REDACTED>
Sent: Wednesday, October 27, 2004 7:32 AM
Subject: Re: How to get configuration data to a large number of threads?


>
>    How about some lateral thinking here:
>
> > I have a transaction based system that spawns a
> > thread to handle every incoming transaction.  Unfortunately there is
> > quite a large chunk of relatively static configuration information that
> > is needed by every thread....
> > The question is, how do I get this information to every thread without
> > significantly slowing things down or using up all the available memory?
>
>
>    You don't - you move the functions to the data - not the data to the
> functions :-)
>
>    Assume the  configuration data is HUGE  (GBytes) - if  this case I'd
> send the computations to  the configuration data. Exactly the opposite
> of what you did :-)
>
>    So you make a global server like this
>
>    -module(config).
>
>    start() ->
>   AHugeDataStructure = read_initiaal_config_data(),
>   loop(AHugeDataStructure).
>
>
>    loop(AHugeDataStructure) ->
>   receive
>      {upgrade, AnotherHugeDataStructure} ->
>   loop(AnotherHugeDataStruce);
>      {compute, From, Fun} ->
>   Val = (catch Fun(AHugeDataStructure)),
>           From ! {self(), Val},
>           loop(AHugeDataStructure)
>   end.
>
> ... now about the clients ...
>
>     Suppose the client wants a username and password
>
>     Make a query
>
>   Query = fun(Config) ->
>      {get_from_config(username, Config),
>       get_from_config(password, Config)
>   end,
>
>    Send it to the config process with a
>
>   Config ! {compute, self(), Query}
>
>   and wait for the reply
>
>
>    Note that sending Funs in messages (locally) is a very lightweight
> operation (as I have explained before :-)
>
>    Now you possibly have a bottleneck in the server - how to solve this?
>
>    Make several servers with identical copies of the configuration data.
>
>    Remember - it might be cheaper to move the functions to the data
> than moving the data to the functions.
>
>    RPC style programming in "Other Languages" (TM) makes you believe
> that there is only only way of doing RPCs (ie moving the data to the
functions)
>
>    Erlang offers freedom of choice.
>
>    Cheers
>
> /Joe
>
>
>
>
>
>