[erlang-questions] Running a fleet of OS processes

[Evadne Wu]

It sounds a lot like mod_cgi so it depends on how much of Erlang/OTP you really need

Also depending on how long the process takes to start, you can start with a pool of one-shot processes and replace them with new ones as you go. I have seen certain cases where a long-running but unused OS process actually decays into a bad state.

IMO the child should report its state instead of the parent having to poll.

As to limiting resource use, I think you can use cgroups. It allows you to set usage limits for Memory, Disk, etc. Once you have that sorted you can then start to limit the number of child OS processes and you can over-provision if necessary.

Regards
Evadne

> On 9 Jul 2018, at 15:33, Yevhenii Kurtov <yevhenii.kurtov@REDACTED> wrote:
> 
> Hi,
> 
> I'm in a situation where most straightforward way to solve the task would be to run a 3rd party binary per each active user. 
> Each of those binaries speaks to the outside world and maintains their own state which is of great importance for the system. Without them user is basically blocked locked.
> 
> The scope of the task is to:
> - start 3rd party program for all users during application launch
> - start 3rd party program on the go for new users
> - continiously poll their state (JSON RPC API) and report updated state if it changed since last poll
> - restart individual apps when they crash
> 
> Idea is to start as many apps as node can carry on and I'm not sure what is the best way to tame this beast.
> 
> I'm surely will use Erlexec and probably will create a gen_server that will keep a list of the processes in its state.
> Then there will be a pool of workers that will go and poll a fleet once in a while.
> 
> That's kinda basic scenario that I can think of, but maybe there are people who had similar problems and can suggest a better approach or advice on how to avoid bottlenecks.
> 
> Apart from solving the architecture there is a resource monitoring problem to not overload the node. The memory utilization is not an issue, but I/O is.  I wonder what would a right marker to stop spawning new processes? Logically I should somehow measure I/O utilization during poll and act accordingly.
> With large enough pool of processes poll should be going all the time :)
> 
> That's basically it. Any input is appreciated.
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions