[erlang-questions] Running a fleet of OS processes

[Yevhenii Kurtov]

Hi,

I'm in a situation where most straightforward way to solve the task would
be to run a 3rd party binary per each active user.
Each of those binaries speaks to the outside world and maintains their own
state which is of great importance for the system. Without them user is
basically blocked locked.

The scope of the task is to:
- start 3rd party program for all users during application launch
- start 3rd party program on the go for new users
- continiously poll their state (JSON RPC API) and report updated state if
it changed since last poll
- restart individual apps when they crash

Idea is to start as many apps as node can carry on and I'm not sure what is
the best way to tame this beast.

I'm surely will use Erlexec and probably will create a gen_server that will
keep a list of the processes in its state.
Then there will be a pool of workers that will go and poll a fleet once in
a while.

That's kinda basic scenario that I can think of, but maybe there are people
who had similar problems and can suggest a better approach or advice on how
to avoid bottlenecks.

Apart from solving the architecture there is a resource monitoring problem
to not overload the node. The memory utilization is not an issue, but I/O
is.  I wonder what would a right marker to stop spawning new processes?
Logically I should somehow measure I/O utilization during poll and act
accordingly.
With large enough pool of processes poll should be going all the time :)

That's basically it. Any input is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20180709/11b5952b/attachment.htm>