[erlang-questions] How much load can supervisors handle?

[Chris Hicks]

> Date: Thu, 25 Oct 2012 20:08:33 +0200
> Subject: Re: [erlang-questions] How much load can supervisors handle?
> From: erlang@REDACTED
> To: silent_vendetta@REDACTED
> CC: erlang-questions@REDACTED
> 
> On Wed, Oct 24, 2012 at 11:18 PM, Chris Hicks
> <silent_vendetta@REDACTED> wrote:
> > Hello folks,
> >
> > I'm currently in the process of building an application which is going to be
> > having a large amount of workers spinning up/down with large variations in
> > the numbers of workers per second. At the high end it could be as many as
> > 10k workers spawning per second, some being long-lived while the majority
> > just do some work and then die, and at the low end just a couple hundred.
> > This work could also be done by using a dynamically sized worker pool, but
> > in either case my primary question is this: How much load can one supervisor
> > handle?
> 
> What do you want the supervisor to do when a process dies?
> 
> 
> /Joe

TL;DR: Not much
Well, there are a couple of scenarios really. One is a static number of supervisors, with the number tuned based on testing of how much load will be generated by the system, and in that case the supervisors won't be doing anything other than the typical house cleaning demanded by the VM. Second scenario is a more dynamic tree of supervisors, which grows/shrinks based on current load, and in that case each supervisor would update some values (either just by sending a message to a monitoring process or updating a table somewhere directly based on the supervisor checking the number of children) and other parts of the system would handle distributing load, starting/stopping supervisors and migrating workers. Antwist on the second scenario would be to not have the supervisors do any of the updates themselves, but have a process polling the supervisors every X seconds and handling the updating itself, in which case the supervisors would be back to just doing the housecleaning the VM demands and responding to periodic requests about it's children.
Chris. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121025/01c5d883/attachment.htm>