[erlang-questions] Distributed Process Spawner

[Jesper Louis Andersen]

In addition to the already mentioned, Game Analytics did a scheduler called
gascheduler:

https://github.com/GameAnalytics/gascheduler

which I haven't tried out or read the source code of. But it does sound
like a fit for the problem at hand.


On Wed, Sep 2, 2015 at 7:20 PM, Michael Truog <mjtruog@REDACTED> wrote:

> On 09/02/2015 04:52 AM, Alex Hudich wrote:
>
>> Hi!
>>
>> I am working on a piece of software which requires start of massive
>> amount of workers (using spawn_link) but now I face with the limitation of
>> one server for handling all the workers.
>> All workers start with simple spawn_link then do their work and return
>> some piece of data to a caller by sending a message.
>>
>> I wonder if I can unload master server and add some additional erlang
>> nodes to perform this job. Of course I looked into pool and slave modules
>> but it seems (beside of setting up ssh keys) does not support dynamic
>> adding or deleting of nodes.
>> Actually I want to know how to build some node supervising module that
>> allows incoming connections from a new slave nodes and automatically plug
>> them into some pool of nodes (including master) for spawning processes.
>> Also it has to exclude down nodes and load balancing workers amongst nodes
>> (so it has to collect some load statistics from nodes)
>>
>>
>> Could you guys help me to find such component or give me advice how to
>> build such thing?
>>
>> Thank you.
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions@REDACTED
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>> If you use the pools pg2 (http://www.erlang.org/doc/man/pg2.html) or cpg
> (https://github.com/okeuday/cpg/)
> and then use node auto-discovery with something like
> https://github.com/okeuday/nodefinder/ .  That can provide
> a way of distributing traffic among processes selected from the pool as
> nodes are added.  CloudI uses cpg with nodefinder
> to make this work with CloudI services, and you could use the same
> functionality with the cloudi_core dependency
> (https://github.com/CloudI/cloudi_core/) due to the cloudi_service_api
> module supporting the API at
> http://cloudi.org/api.html#CloudI .
>
> Load balancing among nodes is done internally due to the pool lookup.  For
> monitoring the load to add or remove nodes
> you could add some basic monitoring.
>
> Best Regards,
> Michael
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
>



-- 
J.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150902/c3d37ee4/attachment.htm>