[erlang-questions] lowering jitter: best practices?

[Felix Gallo]

For a game server, I have a large number (~3000) of concurrent gen_fsm
processes which need to send out UDP packets every 30 milliseconds.  Each
individual UDP-sending function can be presumed to be very short (<1ms) and
not dominate or otherwise affect the scheduling.

I've tested a number of different scenarios:

1.  Each gen_fsm schedules itself via the gen_fsm timeout mechanism.  This
is the easiest and most natural way, but jitter can be +7ms in the 95%
case, and I occasionally see unusual events (e.g. timeout event happens
when only 25-28ms of real time have elapsed, despite 30ms being scheduled).


2.  One gen_fsm 'god timer' process schedules itself and sends out messages
to all of the concurrent gen_fsms to trigger them to send out their UDP
packets upon its own timeout.  Jitter is more variable, probably because
the BEAM decides that the god timer is being too chatty, and sometimes the
gen_fsms overwhelm the god timer in the schedulers.

3.  One port 'god timer' process (written in C, emitting a byte every
30ms).  Jitter is significantly reduced over #2 because apparently port
processes get better scheduling and BEAM doesn't get as angry at the
chattiness.  About on par with #1, maybe a little better, but the design is
unpretty owing to the increased complexity.

Additionally, I've heard of someone using a c-written port-process
god-timer per-fsm, which sounds to me like it would have the absolute best
latency scenario but would involve several thousand external processes, all
competing with erlang for thread execution, which sounds even unprettier.
Haven't tested that one.

Ideally I'd get as close to isochronous as possible, because the game
'feels' right if the stream of incoming packets is uniform.

None of the solutions gets me quite where I wanted; it's possible that any
of the above are the optimal available path and I just don't know that yet,
but I'm wondering if anyone has traveled down this same road and has
advice, tuning secrets, or other plans to share.

F.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20150526/c11accf0/attachment.htm>