[erlang-questions] Maximum number of children for simple_one_for_one supervisor?

[Dmitry Kolesnikov]

Hello,

Yes, the main limitation factor is HW resources. 1M process can be handled both by supervisor and VM. Of course, if process does heavy lifting then you are in trouble. I've made very simple empirical experiment. 
I've used simple_one_to_one supervisor to spawn 1M processes. Each process is container for data and does not perform any actions. First I've wrote 1KB of binary data to each process, there was about 100 writer. After that I was reading this data by 10, 100, and 1000 readers.  Write operation took 100% CPU utilization and beam weight is 4.25GB.
 
You can see some KPI of that experiment from list below.

put by 100 process 1000000 times
-------------------------------------------------------------------------------
Wclock (ms):            233895.186
Completed:              1000000
Failed:                 0 (0.00%)
TPS:                    4275.4
Latency (ms):           0.234
IO tput (MB):           4.192


get by 10 process 10000 times
-------------------------------------------------------------------------------
Wclock (ms):            191.787
Completed:              10000
Failed:                 0 (0.00%)
TPS:                    52141.2
Latency (ms):           0.019
IO tput (MB):           51.118


get by 100 process 100000 times
-------------------------------------------------------------------------------
Wclock (ms):            1455.720
Completed:              100000
Failed:                 0 (0.00%)
TPS:                    68694.5
Latency (ms):           0.015
IO tput (MB):           67.347


get by 1000 process 1000000 times
-------------------------------------------------------------------------------
Wclock (ms):            14794.269
Completed:              1000000
Failed:                 0 (0.00%)
TPS:                    67593.7
Latency (ms):           0.015
IO tput (MB):           66.267

- Dmitry

On Sep 4, 2012, at 5:43 PM, Jesper Louis Andersen wrote:

> On Sep 4, 2012, at 3:42 PM, Son Tran-Nguyen <esente@REDACTED> wrote:
> 
>> My question is how many event managers can the supervisor manage? My goal may
>> get to million of children. I see that I can create millions of processes, but
>> not sure whether the supervisor can manage it effectively.
> 
> My guess would be you will run into limitations long before that. A gentle guess is that a process takes 500 bytes. One million of those is thus half a gigabyte. The supervisor can manage this number, but of course it will use more CPU time and memory managing this large amount by itself.
> 
> My best advice is: Try it and see. We don't know enough about what you are trying to build, so we cannot estimate what will work for you. You will need to set up some hard numbers and then work towards that goal. Also, what hardware do you have available? And do you plan to run distribution? Answering these questions might help you going further.
> 
> You should be able to write an attempt in a relatively short time frame. Just try spawning a million children like the ones you are seeking to run and see if that works for you.
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20120904/9efe9596/attachment.htm>