10 Process Management Optimizations
10.1 Problems
Early versions of the SMP support for the runtime system completely relied on locking in order to protect data accesses from multiple threads. In some cases this isn't that problematic, but in some cases it really is. It complicates the code, ensuring all locks needed are actually held, and ensuring that all locks are acquired in such an order that no deadlock occur. Acquiring locks in the right order often also involve releasing locks held, forcing threads to reread data already read. A good recipe for creation of bugs. Trying to use more fine-grained locking in order to increase possible parallelism in the system makes the complexity situation even worse. Having to acquire a bunch of locks when doing operations also often cause heavy lock contention which cause poor scalability.
Management of processes internally in the runtime system suffered from these problems. When changing state on a process, for example from waiting to runnable, a lock on the process needed to be locked. When inserting a process into a run queue also a lock protecting the run queue had to be locked. When migrating a process from one run queue to another run queue, locks on both run queues and on the process had to be locked.
This last example is a quite common case in during normal operation. For example, when a scheduler thread runs out of work it tries to steal work from another scheduler threads run queue. When searching for a victim to steal from there was a lot of juggling of run queue locks involved, and during the actual theft finalized by having to lock both run queues and the process. When one scheduler runs out of work, often others also do, causing lots of lock contention.
10.2 Solution
Process
In order to avoid these situations we wanted to be able to do most of the fundamental operations on a process without having to acquire a lock on the process. Some examples of such fundamental operations are, moving a process between run queues, detecting if we need to insert it into a run queue or not, detecting if it is alive or not.
All of this information in the process structure that was needed by these operations was protected by the process status lock, but the information was spread across a number of fields. The fields used was typically state fields that could contain a small number of different states. By reordering this information a bit we could easily fit this information into a 32-bit wide field of bit flags (only 12-flags were needed). By moving this information we could remove five 32-bit wide fields and one pointer field from the process structure! The move also enabled us to easily read and change the state using atomic memory operations.
Run Queue
As with processes we wanted to be able to do the most fundamental operations without having to acquire a lock on it. The most important being able to determine if we should enqueue a process in a specific run queue or not. This involves being able to read actual load, and load balancing information.
The load balancing functionality is triggered at repeated fixed intervals. The load balancing more or less strives to even out run queue lengths over the system. When balancing is triggered, information about every run queue is gathered, migrations paths and run queue length limits are set up. Migration paths and limits are fixed until the next balancing has been done. The most important information about each run queue is the maximum run queue length since last balancing. All of this information were previously stored in the run queues themselves.
When a process has become runnable, for example due to reception of a message, we need to determine which run queue to enqueue it in. Previously this at least involved locking the run queue that the process currently was assigned to while holding the status lock on the process. Depending on load we sometimes also had to acquire a lock on another run queue in order to be able to determine if it should be migrated to that run queue or not.
In order to be able to decide which run queue to use without having to lock any run queues, we moved all fixed balancing information out of the run queues into a global memory block. That is, migration paths and run queue limits. Information that need to be frequently updated, like for example maximum run queue length, were kept in the run queue, but instead of operating on this information under locks we now use atomic memory operations when accessing this information. This made it possible to first determine which run queue to use, without locking any run queues, and when decided, lock the chosen run queue and insert the process.
Fixed Balancing Information
When determining which run queue to choose we need to read the fixed balancing information that we moved out of the run queues. This information is global, read only between load balancing operations, but will be changed during a load balancing. We do not want to introduce a global lock that needs to be acquired when accessing this information. A reader optimized rwlock could avoid some of the overhead since the data is most frequently read, but it would unavoidably cause disruption during load balancing, since this information is very frequently read. The likelihood of a large disruption due to this also increase as number of schedulers grows.
Instead of using a global lock protecting modifications of this information, we write a completely new version of it at each load balancing. The new version is written in another memory block than the previous one, and published by issuing a write memory barrier and then storing a pointer to the new memory block in a global variable using an atomic write operation.
When schedulers need to read this information, they read the pointer to currently used information using an atomic read operation, and then issue a data dependency read barrier, which on most architectures is a no-op. That is, it is very little overhead getting access to this information.
Instead of allocating and deallocating memory blocks for the different versions of the balancing information we keep old memory blocks and reuse them when it is safe to do so. In order to be able to determine when it is safe to reuse a block we use the thread progress functionality, ensuring that no threads have any references to the memory block when we reuse it.
Be Less Aggressive
We implemented a test version using lock free run queues. This implementation did however not perform as good as the version using one lock per run queue. The reason for this was not investigated enough to say why this was. Since the locked version performed better we kept it, at least for now. The lock free version, however, forced us to use other solutions, some of them we kept.
Previously when a process that was in a run queue got suspended, we removed it from the queue straight away. This involved locking the process, locking the run queue, and then unlinking it from the double linked list implementing the queue. Removing a process from a lock free queue gets really complicated. Instead, of removing it from the queue, we just leave it in the queue and mark it as suspended. When later selected for execution we check if the process is suspended, if so just dropped it. During its time in the queue, it might also get resumed again, if so execute it when it get selected for execution.
By keeping this part when reverting back to a locked implementation, we could remove a pointer field in each process structure, and avoid unnecessary operations on the process and the queue which might cause contention.
Combined Modifications
By combining the modifications of the process state management and the run queue management, we can do large parts of the work involved when managing processes with regards to scheduling and migration without having any locks locked at all. In these situations we previously had to have multiple locks locked. This of course caused a lot of rewrites across large parts of the runtime system, but the rewrite both simplified code and eliminated locking at a number of places. The major benefit is, of course, reduced contention.
A Benchmark Result
When running the chameneosredux benchmark, schedulers frequently run out of work trying to steal work from each other. That is, either succeeding in migrating, or trying to migrate processes which is a scenario which we wanted to optimize. By the introduction of these improvements, we got a speedup of 25-35% when running this benchmark on a relatively new machine with an Intel i7 quad core processor with hyper-threading using 8 schedulers.