[erlang-questions] is anyone else experiencing reliability issues with R15?

[Rickard Green]

This mail isn't aimed at anyone specific, but more of a general statement.

Loss of performance of an application or bad behavior of an application when going up to a newer release, do not imply that there is a problem with the new release. The only conclusion that can be made is that this combination doesn't work well. It might be a problem with the new release, but it might also be a problem with the application. As an example, any arbitrary optimization improving performance, may trigger a logical bug in the application due to different timing, or overwhelm a consumer due to faster producers in the absence of flow control, etc. Another example, if performance critical parts of the application makes use of functionality that we at OTP do not see as performance critical you may also end up in trouble. I don't know how big of an issue this later example is in reality, but I think I have to say some words about it.

When optimizing the system we sometimes choose to degrade performance of functionality that is not performance critical in order to gain performance of performance critical functionality, or gain overall performance. If  you've made use of such functionality in the critical path, you'll lose overall performance of your application. It may also cause memory issues. When working with scalability improvements this seems to be choices that we have to make more and more often. I suspect we need to be better at informing about such changes.

One such change that I realized isn't mentioned anywhere is the implementation of erlang:memory(). In R15 this call is much more heavyweight from the callers perspective, but from an overall performance perspective much more lightweight.

Preferably these kind of changes wont come as surprises. It is, however, hard to give an exhaustive answer to the question of what we do and do not consider as performance critical functionality, and I will not try to do that. I think common sense will get you far. We do consider core functionality of the language, such as for example message passing, as performance critical. Functionality that pulls out miscellaneous information about the internal state of something is typically not considered as performance critical. Apart from erlang:memory(), process_info() is a good example of such functionality.

Anyhow, if you get into trouble when you upgrade to a new release, you need to find out why. It might be an issue with the release, but it might also very well be an issue with your application. In some cases we might be able to help, but in some cases not.  We are not that many people working here at OTP. The more relevant information you provide, the better chance of getting help. We are also very interested in finding potential issues with OTP, but as I already said we have limited resources.

Regarding increased CPU utilization in R15. When schedulers run out of work, they busy wait for a while before going to sleep. Waking up a busy waiting thread is much faster than waking up a sleeping thread. Due to the rewrites of memory allocation in R15, schedulers are more frequently woken, which cause more busy wait, which in turn cause an increase in CPU utilization when schedulers frequently run out of work (you will at least see some decrease of CPU utilization due to this in R16). When not running out of work there will be no busy wait at all. That is, the increase in CPU utilization does not translate into loss of performance. The busy waiting is there since it shortens the average time to wake up a scheduler, and by this reduces average communication latency between processes. Depending on application the reduced latency might also translate into improved throughput. If the increase in CPU utilization is unwanted, one can as of R15B02 shorten the busy wait threshold (+sbwt command line argument). Note that by shortening the busy wait threshold, there will be an increased average latency.

Regards,
Rickard Green, Erlang/OTP, Ericsson AB