[erlang-questions] Making erlang more realtime?
Mon Mar 6 01:46:19 CET 2017
> If it doesn't, you can try tuning the system in order to meet that requirement: do not run in the cloud, virtualized, eliminte other processes on the machine. Force the CPU clock cycle to be the same all the time.> Do not sleep the CPU. Turn off the NIC so you don't get interference on the memory bus. Replace the OS scheduler with one that gives guaranteed schedules, and so on.
I started doing this to hunt down bottlenecks, I found issues all throughout my stack.
I am going to try again sometime soon. Also I want to try tuning the kernel params for lower latency, I think now that its not Erlangs fault at all here and Erlang can be quite performant down to ~1ms consistent.
The send_after delayed timer resolution issues I was having I think are related to the kernel; proper isolation of erlang runtime from the rest of the OS and scheduler pinning I think will make erlang much more realtime.
On Saturday, March 4, 2017 4:42 PM, Jesper Louis Andersen <jesper.louis.andersen@REDACTED> wrote:
On Thu, Mar 2, 2017 at 9:17 PM Vans S <vans_163@REDACTED> wrote:
I currently need to stream 60 frames per second of video and I made a simple test case using erlang.
Erlang is in many ways a "best effort" realtime system. It tries very hard to deliver timers and handle deadlines quickly, but it has no built-in provisioning to guarantee that they will be delivered on time.
Some systems, which people call "hard" realtime systems, work by you requesting a certain periodic schedule. For instance: I want to be called every 17ms and I will spend at most 2ms of processing time. The system then either accepts that schedule or rejects it based on current workload. Such a system can still fail: CPUs can go bad, and even if you run multiple CPUs in unison and vote on the correct result, the whole system can go bad due to too many failures.
Erlangs best-effort delivery can go bad because of numerous reasons:
* You requested 1,000,000 processes to be woken up on the same point in time. Some will have to wait and come later.* You run on virtualized hardware, and some other task got the CPU.* You are on battery and not AC on your laptop and the system runs at different clock rates.* You are running OSX, which recently started saving power by piggybacking several wakeups together.* You enabled another CPU, which has to become online and then filled with the relevant data before it can process.
Not all of these are the fault of the Erlang subsystem. In general, the only way to get a timely deadline is to overprovision the system enough that it has lots of free time to do what it wants.
That said, I'll say the same as Dmytro, Max, Ferenc and Ilya: try to build your system such that it starts processing as soon as a frame arrives and push it onwards. In a certain sense, what you should be worried about is the frame time, not the frame delay. That is, you should worry about how long it takes to process a frame, because if you start falling behind there, you are in trouble. You may find you don't need utter precision, but can extrapolate a bit around the actual point in time you were woken up. This is where some buffering can be nice. Many computer games toy with this idea: they create the illusion of a 16.7ms frame time, but internally things run on completely different schedules, and it runs with lag: whenever you have a frame in the graphics card buffer, the next one is being painted. This introduces one frame of "lag", but humans are able to learn how to ignore that when they frag people in CS:GO :P
In short: you are in principle requesting that Erlang does something it cannot do in its current form.
You can set up an SLA and then verify that the Erlang system keeps within that SLA. e.g., "for my system, timer delivery happens in the same millisecond that I requested 99.999% of the time". If it doesn't, you can try tuning the system in order to meet that requirement: do not run in the cloud, virtualized, eliminte other processes on the machine. Force the CPU clock cycle to be the same all the time. Do not sleep the CPU. Turn off the NIC so you don't get interference on the memory bus. Replace the OS scheduler with one that gives guaranteed schedules, and so on.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the erlang-questions