[erlang-questions] Distributed Process Spawner
Jesper Louis Andersen
Wed Sep 2 20:19:03 CEST 2015
In addition to the already mentioned, Game Analytics did a scheduler called
which I haven't tried out or read the source code of. But it does sound
like a fit for the problem at hand.
On Wed, Sep 2, 2015 at 7:20 PM, Michael Truog <mjtruog@REDACTED> wrote:
> On 09/02/2015 04:52 AM, Alex Hudich wrote:
>> I am working on a piece of software which requires start of massive
>> amount of workers (using spawn_link) but now I face with the limitation of
>> one server for handling all the workers.
>> All workers start with simple spawn_link then do their work and return
>> some piece of data to a caller by sending a message.
>> I wonder if I can unload master server and add some additional erlang
>> nodes to perform this job. Of course I looked into pool and slave modules
>> but it seems (beside of setting up ssh keys) does not support dynamic
>> adding or deleting of nodes.
>> Actually I want to know how to build some node supervising module that
>> allows incoming connections from a new slave nodes and automatically plug
>> them into some pool of nodes (including master) for spawning processes.
>> Also it has to exclude down nodes and load balancing workers amongst nodes
>> (so it has to collect some load statistics from nodes)
>> Could you guys help me to find such component or give me advice how to
>> build such thing?
>> Thank you.
>> erlang-questions mailing list
>> If you use the pools pg2 (http://www.erlang.org/doc/man/pg2.html) or cpg
> and then use node auto-discovery with something like
> https://github.com/okeuday/nodefinder/ . That can provide
> a way of distributing traffic among processes selected from the pool as
> nodes are added. CloudI uses cpg with nodefinder
> to make this work with CloudI services, and you could use the same
> functionality with the cloudi_core dependency
> (https://github.com/CloudI/cloudi_core/) due to the cloudi_service_api
> module supporting the API at
> http://cloudi.org/api.html#CloudI .
> Load balancing among nodes is done internally due to the pool lookup. For
> monitoring the load to add or remove nodes
> you could add some basic monitoring.
> Best Regards,
> erlang-questions mailing list
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions