SV: How to get configuration data to a large number of threads?
Wed Oct 27 03:27:49 CEST 2004
A practical suggestion for developing and testing different approaches.
1) Make a config module separate from other code.
2) Define a function API for access to the config data.
3) Now you can compare options:
a) make the config data a single big record
b) make the config data a single big binary
c) make the config data a record with several big binaries
a is your worst fear. It allows you to measure the point of memory
b is used if the configuration is not partitionable (i.e., all
transactions need access to most of the configuration)
c is used if the configuration has subsets that are used by particular
4) Create an external API to the config module
a) Allow get and put of config data using 3a, 3b or 3c
b) Config load / update functions for reconfiguration
5) Spawn a process that implements 4a as messages
a) Spawn parameter determines 3a, 3b or 3c as config format
6) Use server messaging to access config from application code
a) Send message to #5 to get the entire config blob needed
b) Use function calls from #2 to obtain values to use
c) Allow variables of 6b to go out of scope quickly to garbage
The above design gives you the flexibility to test various configuration
formats using a config server process. You will only save memory by
using large binaries (bigger than 28 bytes or so in each chunk if using
option #3c) _and_ running process #5 on the same CPU node with shared
heap turned on.
If you want to try embedding the configuration data in each application
process, skip step #5 and #6. Instead pass the config object from #3
when spawning the application process and use the API as before in #6.
More information about the erlang-questions