[erlang-questions] hibernate/timeout
Damien Morton
dmorton@REDACTED
Sat Jun 16 14:55:16 CEST 2007
Yes, I am running Windows.
> These figures were considerably worse than the ones I got
> both on Solaris and Linux. Are you running on Windows?
>
> This is really interesting. We should perhaps take this back
> to the list. (:
>
> BR,
> Ulf W
>
> 2007/6/16, Damien Morton <dmorton@REDACTED>:
>> Ok,
>>
>> I modified the ring:ramp function to start at 10000 and double up until
>> it reaches max_processes.
>>
>> Without a "after 60000" in the recieve_loop, I get results like this:
>>
>> 10000 1.60000 1.00000e-4 1.60010
>> 10000 3.09970 1.00000e-4 3.09980
>> 10000 3.19970 1.00000e-4 3.19980
>> 10000 1.49970 1.59990 3.09960
>> 20000 1.54995 0.799950 2.34990
>> 20000 2.34995 5.00000e-5 2.35000
>> 20000 2.29985 0.799950 3.09980
>> 20000 1.54995 0.799950 2.34990
>> 40000 1.94998 0.399975 2.34995
>> 40000 1.95000 0.399975 2.34998
>> 40000 2.34998 0.374975 2.72495
>> 40000 2.34998 0.399975 2.74995
>> 80000 1.94999 0.587488 2.53748
>> 80000 1.94999 0.587488 2.53748
>> 80000 2.14999 0.587488 2.73747
>> 80000 2.14999 0.587488 2.73747
>> 160000 1.84999 0.587494 2.43749
>> 160000 2.04999 0.587494 2.63749
>> 160000 2.14999 0.587494 2.73749
>> 160000 2.04999 0.581244 2.63124
>> 320000 2.29687 0.584372 2.88124
>> 320000 2.10000 0.587497 2.68749
>> 320000 2.10000 0.584372 2.68437
>> 320000 2.10000 0.584372 2.68437
>>
>> With the "after 60000" I get this:
>>
>> 10000 1.60000 18.6999 20.2999
>> 10000 4.69990 10.9999 15.6998
>> 10000 7.79990 6.19990 13.9998
>> 10000 10.9999 4.69990 15.6998
>> 20000 7.79995 14.8500 22.6499
>> 20000 8.59995 10.1500 18.7499
>> 20000 9.34995 7.79995 17.1499
>> 20000 10.1999 6.24995 16.4499
>> 40000 6.62498 16.4000 23.0250
>> 40000 8.22498 9.37497 17.6000
>> 40000 9.37497 7.39998 16.7750
>> 40000 10.1750 5.84997 16.0250
>> 80000 6.83749 16.2125 23.0500
>> 80000 8.98749 8.97499 17.9625
>> 80000 10.1625 6.24999 16.4125
>> 80000 12.3000 5.27499 17.5750
>> 160000 7.51874 15.9187 23.4375
>> 160000 9.56874 8.59374 18.1625
>> 160000 10.7437 6.34999 17.0937
>> 160000 12.0125 5.26874 17.2812
>> 320000 7.56875 15.7219 23.2906
>> 320000 9.66875 8.49687 18.1656
>> 320000 11.2781 6.15312 17.4312
>> 320000 13.0375 4.88125 17.9187
>>
>> Looks like the timer wheel gets pretty costly for large numbers of
>> processes - interesting that the results are so erratic - with the
>> process creation times getting larger for each run in a given size band,
>> and the send times going down within the band. I tried larger number of
>> runs within each size band, and the creation times keep going up while
>> the send times keep going down with each consecutive run. Weird.
>>
>> Ive attached the modified ring.erl file - perhaps you can run it on your
>> machine. Id be interested to hear what results you get on a linux/unix
>> box with Hipe.
>>
>>
>>
>>
>>
>> > Since I had a ring benchmark lying around, I tested
>> > this up to 300,000 processes. I could detect no increase
>> > in spawn-send cost when increasing from 10,000 to 300,000
>> > processes, where each process entered receive...after,
>> > and all received a message within the prescribed time.
>> > It did seem to cost a few % more per process than
>> > using receive without a timeout.
>> >
>> > I've attached the file. It's easy enough to modify it
>> > by adding an 'after' clause in the receive_loop/2 function
>> > near the end.
>> >
>> > Run e.g. ring:ramp(File, From, To, Step) for both
>> > versions, making sure to start erlang with erl +P N
>> > where N is larger than the largest number of processes
>> > you're trying to create.
>> >
>> > BR,
>> > Ulf W
>> >
>> > 2007/6/15, Ulf Wiger <ulf@REDACTED>:
>> >> We tested the erlang:send_after/3 BIF with some 50,000
>> >> simultaneous timers, and it didn't even blink. I'm not at
>> >> all sure that timer:send_after/3 would be better for any number
>> >> of timers, but I don't know of anyone who's really put this to the
>> test.
>> >>
>> >> One way to test the timer wheel would be to modify a ring benchmark
>> >> so that every process enters a receive ... after ... end clause
>> >> instead of
>> >> the usual receive ... end. You can then run both versions and see if
>> >> there's a noticable difference between the two, and if you can detect
>> >> and scalability problem with the version where every process in
>> effect
>> >> starts and stops a timer.
>> >>
>> >> BR,
>> >> Ulf W
>> >>
>> >> 2007/6/15, Damien Morton <dmorton@REDACTED>:
>> >> > So it would seem to make sense to put hibernate() into my timeout
>> >> > handler, and to use erlang:send_after for a wakeup call.
>> >> >
>> >> > On #erlang, it was suggested that timer:send_after was better than
>> >> > erlang:send_after when managing very large numbers of processes.
>> >> >
>> >> > I'm planning on having potentially millions of processes, each of
>> >> which
>> >> > will tend to sleep for minutes or even hours - which of
>> >> timer:send_after
>> >> > or erlang:send_after would you recommend?
>> >> > > Hi Damien,
>> >> > >
>> >> > > The hibernate/3 function is really useful when you have
>> >> > > lots of long lived processes, and you need to save memory.
>> >> > >
>> >> > > The timeout clause in the receive is not relevant to
>> >> > > hibernate, because you cannot hibernate and wait in
>> >> > > a receive at the same time. You can do a receive with
>> >> > > a timeout, and go into hibernation after a certain time,
>> >> > > but I think it's better to simply hibernate first, if you
>> >> > > suspect that it may be a while before the next message
>> >> > > comes in. One simple way of using it might be to always
>> >> > > hibernate when transitioning into a stable state.
>> >> > >
>> >> > > And, yes, you should use the timer BIFs rather than the
>> >> > > timer module.
>> >> > >
>> >> > > BR,
>> >> > > Ulf W
>> >> > >
>> >> > > 2007/6/14, Damien Morton <dmorton@REDACTED>:
>> >> > >
>> >> > >> Hello,
>> >> > >>
>> >> > >> Im new to the list, so... hello everyone.
>> >> > >>
>> >> > >> Wanted to ask a question about managing large numbers of very
>> >> > >> lightweight processes.
>> >> > >>
>> >> > >> Ive discovered the hibernate() function, which should prove
>> >> usefull as
>> >> > >> most of the processes I will be using will be asleep for
>> extended
>> >> > >> periods of time - minutes, hours and even days, though this
>> >> sleep will
>> >> > >> very often be interrupted.
>> >> > >>
>> >> > >> Im wondering, however, if a process which is hibernating will
>> >> timeout if
>> >> > >> its receive loop has a timeout clause, or whether, once
>> placed into
>> >> > >> hibernation, it will remain there until an external process
>> >> sends it a
>> >> > >> message.
>> >> > >>
>> >> > >> My intention was to make processes hibernate after a period of
>> >> time in
>> >> > >> which they receive no messages, and to then wake up after a
>> certain
>> >> > >> period of time.
>> >> > >>
>> >> > >> Ive had a look at the timer module, which seems to use ets as a
>> >> priority
>> >> > >> queue. Id rather stick with the built-in timeout mechanism, if
>> >> possible.
>> >> > >> >From what I read, its timer-wheel implementation is very
>> >> efficient.
>> >> > >>
>> >> > >> Thanks,
>> >> > >>
>> >> > >> Damien Morton
>> >> > >>
>> >> > >>
>> >> > >
>> >> >
>> >> >
>> >>
>>
>>
>> -module(ring).
>>
>> -export([ramp/0]).
>>
>> ramp() ->
>> process_flag(trap_exit, true),
>> ramp(10000,erlang:system_info(process_limit)-100).
>>
>> ramp(N, Max) when N < Max ->
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> format(do_run(N)),
>> ramp(N*2,Max);
>> ramp(_,_) -> true.
>>
>>
>> do_run(N) ->
>> Pid = spawn_link(fun() ->
>> run(N)
>> end),
>> receive
>> {'EXIT', Pid, Result} ->
>> {N, Result}
>> end.
>>
>> format({N, {Spawn, Send}}) ->
>> io:format("~w\t~w\t~w\t~w~n", [N, Spawn, Send, Spawn+Send]).
>>
>>
>> run(N) when integer(N), N > 1 ->
>> {First, Last, SpawnTime} = spawn_ring(N),
>> SendTime = send_ring(First, Last),
>> exit({SpawnTime/N, SendTime/N}).
>>
>>
>> spawn_ring(N) ->
>> Start = erlang:now(),
>> StarterProcess = self(),
>> First = spawn_child(N, StarterProcess),
>> receive
>> {Last, done} ->
>> SpawnStop = erlang:now(),
>> {First, Last, time_diff(SpawnStop, Start)}
>> after 60000 ->
>> exit(timeout)
>> end.
>>
>> send_ring(First, Last) ->
>> Start = erlang:now(),
>> First ! {self(), ping},
>> receive
>> {Last, ping} ->
>> Stop = erlang:now(),
>> time_diff(Stop, Start)
>> after 60000 ->
>> exit(timeout)
>> end.
>>
>> spawn_child(N, StarterProcess) ->
>> Parent = self(),
>> spawn_link(fun() ->
>> child_init(N, Parent, StarterProcess)
>> end).
>>
>> child_init(1, Parent, StarterProcess) ->
>> StarterProcess ! {self(), done},
>> receive_loop(Parent, StarterProcess);
>> child_init(N, Parent, StarterProcess) ->
>> Next = spawn_child(N-1, StarterProcess),
>> receive_loop(Parent, Next).
>>
>> receive_loop(Parent, Next) ->
>> receive
>> {Parent, ping} ->
>> Next ! {self(), ping},
>> receive_loop(Parent, Next)
>> after 60000 -> exit(timeout)
>> end.
>>
>>
>> time_diff(After, Before) ->
>> %% snipped from timer:tc/3
>> (element(1,After)*1000000000000 +
>> element(2,After)*1000000 +
>> element(3,After)) -
>> (element(1,Before)*1000000000000 +
>> element(2,Before)*1000000 + element(3,Before)).
>>
>>
>
>
More information about the erlang-questions
mailing list