[erlang-questions] Distributed Process Spawner

[Michael Truog]

On 09/02/2015 04:52 AM, Alex Hudich wrote:
> Hi!
>
> I am working on a piece of software which requires start of massive amount of workers (using spawn_link) but now I face with the limitation of one server for handling all the workers.
> All workers start with simple spawn_link then do their work and return some piece of data to a caller by sending a message.
>
> I wonder if I can unload master server and add some additional erlang nodes to perform this job. Of course I looked into pool and slave modules but it seems (beside of setting up ssh keys) does not support dynamic adding or deleting of nodes.
> Actually I want to know how to build some node supervising module that allows incoming connections from a new slave nodes and automatically plug them into some pool of nodes (including master) for spawning processes. Also it has to exclude down nodes and load balancing workers amongst nodes (so it has to collect some load statistics from nodes)
>
>
> Could you guys help me to find such component or give me advice how to build such thing?
>
> Thank you.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>
If you use the pools pg2 (http://www.erlang.org/doc/man/pg2.html) or cpg (https://github.com/okeuday/cpg/)
and then use node auto-discovery with something like https://github.com/okeuday/nodefinder/ .  That can provide
a way of distributing traffic among processes selected from the pool as nodes are added.  CloudI uses cpg with nodefinder
to make this work with CloudI services, and you could use the same functionality with the cloudi_core dependency
(https://github.com/CloudI/cloudi_core/) due to the cloudi_service_api module supporting the API at
http://cloudi.org/api.html#CloudI .

Load balancing among nodes is done internally due to the pool lookup.  For monitoring the load to add or remove nodes
you could add some basic monitoring.

Best Regards,
Michael