[erlang-questions] Timers for hibernated processes

[Zvi]

Yes, this what we did and it works well. We have a pool of N
gen_servers, broadcasting timer messages. Less flexible, but it works.

I suspect, that the problem is either in the new OTP R14B04/03 or in
the Cowboy code. B/c a year ago I was able to create millions of
timers, when our system was based on R13 and Mochiweb.

On Nov 22, 4:29 pm, Loïc Hoguin <es...@REDACTED> wrote:
> Hello,
>
> Why not create a pool of 25 timers, one per second, and then associate
> connections with the right timer so it gets its idle message at the
> right time? This could be done with 250 timers probably without issue
> too, maybe even 2500.
>
> On 11/22/2011 03:20 PM, Zvi wrote:
>
>
>
>
>
> > The problem DOESN'T happen with timers disabled (via macro) or when
> > using only single timer.
>
> > I suspect, that timers are not implemented in scalable way in the BEAM
> > or there are any soft/hard limits (like for number of processes/ports/
> > ets/etc.)
>
> > We monitor message queue length of all of our registered processes and
> > total number of messages waiting in the VM. All queues are either 0 or
> > low numbers and CPU usage is 0% or near zero.
>
> > Thanks for the suggestion  to use eprof - we'll try it.
>
> > On Nov 22, 4:07 pm, Ahmed Omar<spawn.th...@REDACTED>  wrote:
> >> Hi Zvi,
> >> It would be useful if you can use
> >> eprof<http://www.erlang.org/doc/man/eprof.html>to get some
> >> measurements about how time is spent, is it really in starting
> >> the timer or something else.
>
> >> On Tue, Nov 22, 2011 at 1:02 PM, Zvi<zvi.avra...@REDACTED>  wrote:
> >>> Hi,
>
> >>> I'll try to add more info to Ori's post (we work together).
>
> >>> The code of our tests is public:
>
> >>> 1. The version with timer per process (maximum we can get ~ 370K
> >>> concurrent connections out of 500K):
>
> >>>https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>
> >>> 2. The version with single timer - broadcasting idle message to all
> >>> per-connection processes (no problems - we can get 500K  concurrent
> >>> connections out of 500K):
>
> >>>https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>
> >>> We also tried variation of [1] but with hibernate - same results.
> >>> Both demos started using variation of go.sh script:
>
> >>>https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>
> >>> The solution with single timer is not good for us, because the
> >>> broadcast itself may take some time (seconds even using high
> >>> priority). Also we have application-level timers in almost every
> >>> gen_server in  our codebase.
>
> >>> Right now we experimenting with creating our own timer service, which
> >>> will reuse one timer for groups of processes.
>
> >>> So the interesting question is, how timers implemented inside BEAM /
> >>> stdlib and what's effective way to replace them.
> >>> Do timers implemented using processes?
>
> >>> Thanks in advance,
> >>> Zvi
>
> >>> On Nov 22, 12:58 pm, Ahmed Omar<spawn.th...@REDACTED>  wrote:
> >>>> Hi Ori,
> >>>> Can you be a bit more specific how it hurts your performance? Is there
> >>> only
> >>>> one process responsible for creating timers? What does your app do in the
> >>>> first place?
>
> >>>> On Tue, Nov 22, 2011 at 2:32 AM, ori brost<oribr...@REDACTED>  wrote:
> >>>>> I tried hibernation alone and it did not help, only
> >>>>> hibernationcombined with reducing timer amount did the trick
> >>>>> On Mon, Nov 21, 2011 at 9:25 PM, Joel Reymont<joe...@REDACTED>
> >>> wrote:
> >>>>>> Are you sure that creating these timers is what's hurting
> >>> performance?
>
> >>>>>> My bet is on your processes, try hibernating them.
>
> >>>>>> Let us know if that helps.
>
> >>>>>> On Nov 21, 2011, at 7:22 PM, ori brost wrote:
>
> >>>>>>> We have an erlang program which creates many (300000) timers using
> >>>>>>> erlang:start_timer. We have noticed in tests that creating these
> >>>>>>> timers hurts our performance. For each such timer we have a process
> >>>>>>> that gets a message from it once every 25 seconds, except for
> >>> waiting
> >>>>>>> for these messages, the processes are mostly idle. We are working on
> >>>>>>> reducing the number of timers, but we would also like to know
> >>> whether
> >>>>>>> these timers may hurt performance less if the processes would
> >>>>>>> hibernate until they get a message. Furthermore, is there any
> >>>>>>> documentation explaining why creating many timers can hurt erlang
> >>>>>>> performance?
>
> >>> --------------------------------------------------------------------------
> >>>>>> - for hire: mac osx device driver ninja, kernel extensions and usb
> >>>>> drivers
>
> >>> ---------------------+------------+---------------------------------------
> >>>>>>http://wagerlabs.com|@wagerlabs |
> >>>>>http://www.linkedin.com/in/joelreymont
>
> >>> ---------------------+------------+---------------------------------------
>
> >>>>> _______________________________________________
> >>>>> erlang-questions mailing list
> >>>>> erlang-questi...@REDACTED
> >>>>>http://erlang.org/mailman/listinfo/erlang-questions
>
> >>>> --
> >>>> Best Regards,
> >>>> - Ahmed Omarhttp://nl.linkedin.com/in/adiaa
> >>>> Follow me on twitter
> >>>> @spawn_think<http://twitter.com/#!/spawn_think>
>
> >>>> _______________________________________________
> >>>> erlang-questions mailing list
> >>>> erlang-questi...@REDACTED://
> >>> erlang.org/mailman/listinfo/erlang-questions
> >>> _______________________________________________
> >>> erlang-questions mailing list
> >>> erlang-questi...@REDACTED
> >>>http://erlang.org/mailman/listinfo/erlang-questions
>
> >> --
> >> Best Regards,
> >> - Ahmed Omarhttp://nl.linkedin.com/in/adiaa
> >> Follow me on twitter
> >> @spawn_think<http://twitter.com/#!/spawn_think>
>
> >> _______________________________________________
> >> erlang-questions mailing list
> >> erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questi...@REDACTED
> >http://erlang.org/mailman/listinfo/erlang-questions
>
> --
> Loïc Hoguin
> Dev:Extend
> _______________________________________________
> erlang-questions mailing list
> erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions