[erlang-questions] How much load can supervisors handle?

[Chris Hicks]

> <snip>>
> Good - numbers - I like numbers, we can play with numbers.
> 
> You didn't say how long time the 20K workers take to arrive, but no
> matter, we can
> do a little experiment anyway ...
> 
> I made a simple measurements to get a feeling for this (the code follows ...)
> 
> I'll start 20K processes, each process has a 5KB heap,
> and lives forever.
> 
> They are started by a single process - which is a bottleneck - I just want to
> measure the times and memory.
> 
> > test1:test(20000).
> {937304,{150075784,7503}}
> 
> So it took 0.9 seconds (ie 46 micro seconds/process)
> and used 150 Meg of memory - the space per process is 7503 bytes
> 
> This was done on a dual core 2.53 GHz 4 Meg memory machine
> 
> (it might be an idea to hibernate the sleeping processes here - I
> haven't tried this)
> 
> You could parallelize the master routine - to reduce this 0.9 seconds
> since it's a dual core you might get a factor 2 - how many masters you'd need
> needs tweaking. Say 2 times the number of cores. On a dual core try
> 3,4,5 ... masters
> 
> I strongly suspect that these figures tell you nothing and that the
> real problems will be in the tcp overheads - assuming you keep a load of
> sockets open. This is more difficult to measure - you need a cluster to
> load the server to the point where it breaks to measure this.
> 
> Here's the program I wrote for these measurements.
> 
> It gives no more than a ball-park figure to start playing with
> but might give a vague indication of what to expect
> 
> Cheers
> 
> /Joe
> 
> -module(test1).
> 
> -compile(export_all).
> 
> test(N) ->
>     timer:tc(?MODULE,test0,[N]).
> 
> test0(N) ->
>     M1 = proplists:get_value(total, erlang:memory()),
>     Pid = spawn(fun master/0),
>     loop(N, Pid),
>     M2 = proplists:get_value(total, erlang:memory()),
>     {M2-M1, (M2-M1) div N}.
> 
> loop(0,_) -> true;
> loop(N, Pid) ->
>     rpc(Pid, start_child),
>     loop(N-1, Pid).
> 
> rpc(Pid, M) ->
>     Pid ! {self(), M},
>     receive
> 	{Pid, Reply} ->
> 	    Reply
>     end.
> 
> master() ->
>     receive
> 	{From, start_child} ->
> 	    Pid = start_child(),
> 	    From ! {self(), ack},
> 	    master()
>     end.
> 
> start_child() ->
>     spawn(fun child/0).
> 
> child() ->
>     %% 5K Byte word heap
>     %% one list cell = 8 bytes on 32 bit erlang
>     list_to_binary(lists:duplicate(625, 42)),
>     receive
> 	after infinity ->
> 		true
> 	end.

Joe,
You, sir, are a rock star. While I'm certain you could have written that code and tested it, all while asleep, I really appreciate the time you are taking to help me understand more about what I'm trying to accomplish. While, as you point out, these numbers are fairly useless outside of a real running system where the bottlenecks are likely to be elsewhere this certainly does give me a good starting place. Anyway, you've given me several things to think about and I've already got some ideas as to how I'm going to accomplish some of what I want, including a beast of a custom management layer.
I'm sure once I get more of the system built I'll be coming back to check my numbers and ask for suggestions on improving performance. And I'm certain I'll be asking for more advice as I move forward but, until then, I think I've got what I needed for now so I'll yield the floor. Thank you again everyone for your feedback.
Chris. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121026/347e0a3e/attachment.htm>