[erlang-questions] gen_server and multiple workers

[David King]

> You could create multiple gen_servers and instead of registering
> their pids (using start_link/4) instead keep them anonymous (using
> start_link/3) and then have them joined a named process group (using
> pg2:join/2) and then issue requests using pg2:get_closest_pid/1.

That sounds like exactly what I want. Does pg2:get_closest_pid/1  
handle balancing, or will it return the same PID every time it's  
called from a given machine? (The documentation is not clear on  
this.) Is a named process group accessible from any machine in a  
cluster?

Thanks for getting back to me so quickly, by the way!

> On Thu, 30 Aug 2007, David King wrote:
>
>> I imagine that this is a problem that has been solved a hundred times
>> and that I just don't have the right search terms :)
>>
>> I need to communicate with an outside program, for which I'm using a
>> port and a gen_server. However, I may be getting many requests for
>> communication to this port at a time, and I don't want to introduce a
>> bottleneck in the gen_server, nor do I want to fork() a new instance
>> of the program for every request. So I think I'd like to have several
>> workers, where each request grabs an available worker (blocking until
>> one is available), ideally with some place to put logic to expand or
>> reduce the size of the worker-pool with load, but that's optional.
>> Having several copies of the external program open at a time isn't a
>> problem (it's just a text-transform done by a Perl program).
>>
>> My gen_server doesn't keep any state except for the Port, so that
>> could be managed another way. It seems that I can't combine this with
>> gen_server, as gen_server seems to want to register() its PID, and I
>> can't have multiple workers with the same register()ed name.
>>
>> I have a supervisor watching the gen_server. supervisor's required
>> export init/1 returns (among other things) a list of processes to
>> enter the supervision, but it just calls start_link on the
>> gen_server, which registers its name.
>>
>> I could have the worker-pool managed by the single gen_server
>> instance, but I'd like them to be able to crash independently, and it
>> seems that that would complicate the handle_call function (as
>> gen_server seems to assume that it's synchronous). Complication is
>> okay, but I'd like to avoid it if it turns out that there is a single
>> library that already does what I want :)
>>
>> I could also have the supervisor pass in a different name to register
>> to gen_server:start_link (worker1,worker2, ...) and have the
>> supervisor manage the pool, but that seems messy (which, again, is
>> okay, but is to be avoided if possible).
>>
>> Anyway, I'm sure this is a solved problem and I'm just looking in the
>> wrong places. How would you do this?