[erlang-questions] Making erlang more realtime?
Jesper Louis Andersen
Sat Mar 4 22:42:41 CET 2017
On Thu, Mar 2, 2017 at 9:17 PM Vans S <vans_163@REDACTED> wrote:
> I currently need to stream 60 frames per second of video and I made a
> simple test case using erlang.
Erlang is in many ways a "best effort" realtime system. It tries very hard
to deliver timers and handle deadlines quickly, but it has no built-in
provisioning to guarantee that they will be delivered on time.
Some systems, which people call "hard" realtime systems, work by you
requesting a certain periodic schedule. For instance: I want to be called
every 17ms and I will spend at most 2ms of processing time. The system then
either accepts that schedule or rejects it based on current workload. Such
a system can still fail: CPUs can go bad, and even if you run multiple CPUs
in unison and vote on the correct result, the whole system can go bad due
to too many failures.
Erlangs best-effort delivery can go bad because of numerous reasons:
* You requested 1,000,000 processes to be woken up on the same point in
time. Some will have to wait and come later.
* You run on virtualized hardware, and some other task got the CPU.
* You are on battery and not AC on your laptop and the system runs at
different clock rates.
* You are running OSX, which recently started saving power by piggybacking
several wakeups together.
* You enabled another CPU, which has to become online and then filled with
the relevant data before it can process.
Not all of these are the fault of the Erlang subsystem. In general, the
only way to get a timely deadline is to overprovision the system enough
that it has lots of free time to do what it wants.
That said, I'll say the same as Dmytro, Max, Ferenc and Ilya: try to build
your system such that it starts processing as soon as a frame arrives and
push it onwards. In a certain sense, what you should be worried about is
the frame time, not the frame delay. That is, you should worry about how
long it takes to process a frame, because if you start falling behind
there, you are in trouble. You may find you don't need utter precision, but
can extrapolate a bit around the actual point in time you were woken up.
This is where some buffering can be nice. Many computer games toy with this
idea: they create the illusion of a 16.7ms frame time, but internally
things run on completely different schedules, and it runs with lag:
whenever you have a frame in the graphics card buffer, the next one is
being painted. This introduces one frame of "lag", but humans are able to
learn how to ignore that when they frag people in CS:GO :P
In short: you are in principle requesting that Erlang does something it
cannot do in its current form.
You can set up an SLA and then verify that the Erlang system keeps within
that SLA. e.g., "for my system, timer delivery happens in the same
millisecond that I requested 99.999% of the time". If it doesn't, you can
try tuning the system in order to meet that requirement: do not run in the
cloud, virtualized, eliminte other processes on the machine. Force the CPU
clock cycle to be the same all the time. Do not sleep the CPU. Turn off the
NIC so you don't get interference on the memory bus. Replace the OS
scheduler with one that gives guaranteed schedules, and so on.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions