[erlang-questions] Timers for hibernated processes

[Loïc Hoguin]

Hello,

Why not create a pool of 25 timers, one per second, and then associate 
connections with the right timer so it gets its idle message at the 
right time? This could be done with 250 timers probably without issue 
too, maybe even 2500.

On 11/22/2011 03:20 PM, Zvi wrote:
> The problem DOESN'T happen with timers disabled (via macro) or when
> using only single timer.
>
> I suspect, that timers are not implemented in scalable way in the BEAM
> or there are any soft/hard limits (like for number of processes/ports/
> ets/etc.)
>
> We monitor message queue length of all of our registered processes and
> total number of messages waiting in the VM. All queues are either 0 or
> low numbers and CPU usage is 0% or near zero.
>
> Thanks for the suggestion  to use eprof - we'll try it.
>
> On Nov 22, 4:07 pm, Ahmed Omar<spawn.th...@REDACTED>  wrote:
>> Hi Zvi,
>> It would be useful if you can use
>> eprof<http://www.erlang.org/doc/man/eprof.html>to get some
>> measurements about how time is spent, is it really in starting
>> the timer or something else.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Nov 22, 2011 at 1:02 PM, Zvi<zvi.avra...@REDACTED>  wrote:
>>> Hi,
>>
>>> I'll try to add more info to Ori's post (we work together).
>>
>>> The code of our tests is public:
>>
>>> 1. The version with timer per process (maximum we can get ~ 370K
>>> concurrent connections out of 500K):
>>
>>> https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>>
>>> 2. The version with single timer - broadcasting idle message to all
>>> per-connection processes (no problems - we can get 500K  concurrent
>>> connections out of 500K):
>>
>>> https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>>
>>> We also tried variation of [1] but with hibernate - same results.
>>> Both demos started using variation of go.sh script:
>>
>>> https://github.com/nivertech/cowboy/blob/ori_acceptor_rate_throttling...
>>
>>> The solution with single timer is not good for us, because the
>>> broadcast itself may take some time (seconds even using high
>>> priority). Also we have application-level timers in almost every
>>> gen_server in  our codebase.
>>
>>> Right now we experimenting with creating our own timer service, which
>>> will reuse one timer for groups of processes.
>>
>>> So the interesting question is, how timers implemented inside BEAM /
>>> stdlib and what's effective way to replace them.
>>> Do timers implemented using processes?
>>
>>> Thanks in advance,
>>> Zvi
>>
>>> On Nov 22, 12:58 pm, Ahmed Omar<spawn.th...@REDACTED>  wrote:
>>>> Hi Ori,
>>>> Can you be a bit more specific how it hurts your performance? Is there
>>> only
>>>> one process responsible for creating timers? What does your app do in the
>>>> first place?
>>
>>>> On Tue, Nov 22, 2011 at 2:32 AM, ori brost<oribr...@REDACTED>  wrote:
>>>>> I tried hibernation alone and it did not help, only
>>>>> hibernationcombined with reducing timer amount did the trick
>>>>> On Mon, Nov 21, 2011 at 9:25 PM, Joel Reymont<joe...@REDACTED>
>>> wrote:
>>>>>> Are you sure that creating these timers is what's hurting
>>> performance?
>>
>>>>>> My bet is on your processes, try hibernating them.
>>
>>>>>> Let us know if that helps.
>>
>>>>>> On Nov 21, 2011, at 7:22 PM, ori brost wrote:
>>
>>>>>>> We have an erlang program which creates many (300000) timers using
>>>>>>> erlang:start_timer. We have noticed in tests that creating these
>>>>>>> timers hurts our performance. For each such timer we have a process
>>>>>>> that gets a message from it once every 25 seconds, except for
>>> waiting
>>>>>>> for these messages, the processes are mostly idle. We are working on
>>>>>>> reducing the number of timers, but we would also like to know
>>> whether
>>>>>>> these timers may hurt performance less if the processes would
>>>>>>> hibernate until they get a message. Furthermore, is there any
>>>>>>> documentation explaining why creating many timers can hurt erlang
>>>>>>> performance?
>>
>>> --------------------------------------------------------------------------
>>>>>> - for hire: mac osx device driver ninja, kernel extensions and usb
>>>>> drivers
>>
>>> ---------------------+------------+---------------------------------------
>>>>>> http://wagerlabs.com|@wagerlabs |
>>>>> http://www.linkedin.com/in/joelreymont
>>
>>> ---------------------+------------+---------------------------------------
>>
>>>>> _______________________________________________
>>>>> erlang-questions mailing list
>>>>> erlang-questi...@REDACTED
>>>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>>> --
>>>> Best Regards,
>>>> - Ahmed Omarhttp://nl.linkedin.com/in/adiaa
>>>> Follow me on twitter
>>>> @spawn_think<http://twitter.com/#!/spawn_think>
>>
>>>> _______________________________________________
>>>> erlang-questions mailing list
>>>> erlang-questi...@REDACTED://
>>> erlang.org/mailman/listinfo/erlang-questions
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questi...@REDACTED
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>> --
>> Best Regards,
>> - Ahmed Omarhttp://nl.linkedin.com/in/adiaa
>> Follow me on twitter
>> @spawn_think<http://twitter.com/#!/spawn_think>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questi...@REDACTED://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions


-- 
Loïc Hoguin
Dev:Extend