[erlang-questions] Some facts about Erlang and SMP

[Edwin Fine]

Kenneth,

Thank you very much for answering many, if not all, of my questions. It is
extremely useful to get an authoritative answer like yours, upon which one
can base objective performance-related decisions.

For example, as you stated,

>> Of course the running
>> of several VM's require that the application can run
>> in many parallel tasks which has no or very little communication with
>> each other.

If using -smp disable (not +S 1, which I mistakenly thought was equivalent
to -smp disable), there are applications that could benefit considerably,
such as those that have many thousands of concurrent TCP/IP-initiated
processes (what Joe Armstrong calls "naturally concurrent") that are not
related to each other and therefore have no significant cross-VM IPC. A
non-database backed content-serving web site is one example. Partitioning
what would amount to numerous HTTP GET requests across multiple Erlang -smp
disabled VMs, one per core, using a hardware load balancer, is likely to
show performance benefits over the homogeneous SMP approach. Note that this
scenario is consistent with your requirements of (a) little or no cross-VM
IPC and (b) little or no heavy Mnesia or ETS table sharing.

One could potentially make good use of a CPU affinity command such as
taskset (as you mentioned). This would have applicability where, for
example, one wanted to "reserve" some CPUs for non-Erlang usage, such as to
run some special-purpose high-priority application that would suffer
performance-wise if it were to be preempted by the OS. If using -smp
disable, then the single thread of a VM can safely be affinitied to a
specific CPU or core and will not "migrate" to other processors. In
addition, it is arguably possible that in this and similar scenarios,
keeping a VM on a single core will benefit from improved processor and data
cache hit rates and reduced or eliminated cache coherency storms. The part
that is arguable is that if the VM itself is preempted, the contents of the
processor cache may become invalidated anyway. This suggests that the VM
should be run at an elevated priority and be kept busy so that it is always
the top contender for the OS run queue and seldom gets rescheduled.

Of course, this is advanced performance tweaking and will probably be
unnecessary for the majority of Erlang applications. However, for those who
are trying to wring the last iota of performance out of their applications,
this information can be invaluable.

In more common situations, this information can help Erlang-based system
designers avoid lock contention by taking it into account when designing
around database and ETS tables. In some cases it may make sense to have a
process that owns a private table, and have all other processes interact
with the table via this "gatekeeper". Careful measurement and thought will
be needed to establish whether the bottleneck created by serializing access
through an Erlang gen_server-style process is better or worse than the
bottleneck created by serializing access through an ETS table lock, for
example.

One final thought is that whenever some product is invented for a specific
purpose, and excels at that purpose, as it becomes more popular it begins to
get used in ways that its original designers never anticipated. Some of
these uses are in line with the original design intent, but push the
envelope and need improvements to the product implementation, and others are
simply stretching the capabilities of the product in inappropriate
directions. This has been seen in numerous cases, and seems to stem from the
desire to have one tool that can do everything, and not have to know and
support multiple tools. Then we have a hammer, and the world starts to look
like a collection of nails.

As far as I can see, the answer is always the same: use the tool that is
right for the job. Don't fall into the trap of "100% Pure Erlang"
everywhere. It's less convenient, but if you want performance, offload the
heavy crunching stuff to 'C' drivers. For example, instead of using xmerl
for heavy XML processing, do what ejabberd does: use the 'C' expat parser
via a linked-in driver. The Ruby community had exactly the same issue with
XML parsing. Need heavy use of regular expressions? Use re, not regexp. Need
heavy database processing? Maybe Mnesia is not the right tool for this
particular job, although it may excel in the dimension for which it was
designed: soft real-time telecomms applications. And so on. True, it is less
convenient to have to step outside the environment in which one has become
comfortable, and have architecture-specific applications (and risks to the
robustness of the VM if the LID has any crashworthy bugs), but sometimes if
you need the performance, and your algorithms are already appropriate, you
have to use the right tool for the job and take the risks.

As support for this argument, the AXD301 project has "a couple of million"
lines of Erlang code, and at one point had over 1 million lines of C/C++
code. The exact numbers seem to be elusive or outdated (the Erlang FAQ is
way out of date, and other documents I found are vague as to how much C/C++
is currently used); if someone (e.g. Ulf Wiger) could provide more accurate
information I'd appreciate it. How much of the C/C++ code is used for device
interfaces and is hence unavoidable, how much for efficiency considerations,
and how much for other purposes is not known to me. Still, the point is that
this is not a "100% pure Erlang" implementation, and I would be very
surprised if at least some of that C/C++ code was not there for efficiency.

Thank you again.

Regards,
Edwin Fine



On Tue, Sep 16, 2008 at 5:34 AM, Kenneth Lundin <kenneth.lundin@REDACTED>wrote:

> Here are some short facts about how the Erlang SMP implementation
> works and how it
> relates to performance and scalability.
>
> There will be a more detailed description of how multi core works and
> on the future plans available
> in a couple of weeks. I plan to include some of this in my
> presentation at the ICFP2008, Erlang Workshop in Victoria BC,
> September 27
>
> The Erlang VM without SMP support has 1 scheduler which runs in the
> main process thread. The scheduler
> picks runnable Erlang processes and IO-jobs from the run-queue and
> there is no need to lock data structures since
> there is only one thread accessing them.
>
> The Erlang VM with SMP support can have 1 to many schedulers which are
> run in 1 thread each. The schedulers pick runnable Erlang processes
> and IO-jobs from one common run-queue. In the SMP VM all shared data
> structures are
> protected with locks, the run-queue is one example of a data structure
> protected with locks.
>
> >From OTP R12B the SMP version of the VM is automatically started as
> default if the OS reports more than 1 CPU (or Core) and with the same
> number of schedulers as CPU's or Cores.
>
> You can see what was chosen at the first line of printout from the
> "erl" command. E.g.
> Erlang (BEAM) emulator version 5.6.4 [source] [smp:4] [asynch-threads:0]
> .....
>
> The "[smp:4]" above tells that the SMP VM is run and with 4 schedulers.
>
> The default behaviour can be overridden with the
>  "-smp [enable|disable|auto]"  auto is default
> and to set the number of schedulers, if smp is set to enable or auto
> "+S Number"         where Number is the number of schedulers (1..1024)
>
> Note ! that it is normally nothing to gain from running with more
> schedulers than the number of CPU's or Cores.
> Note2 ! On some operating systems the number of CPU's or Cores to be
> used by a process can be restricted
> with commands. For example on Linux the command "taskset" can be used
> for this. The Erlang VM will
> currently only detect number of available CPU's or Cores and will not
> take the mask set by "taskset" into account.
> Because of this it can happen and has happened that e.g. only 2 Cores
> are used even if the Erlang VM
> runs with 4 schedulers. It is the OS that limits this because it take
> the mask from "taskset" into account.
>
> The schedulers in the Erlang VM are run on one OS-thread each and it
> is the OS that decides if the threads are
> executed on different Cores. Normally the OS will do this just fine
> and will also keep the thread on the same Core throughout the
> execution.
>
> The Erlang processes will be run by different schedulers because they
> are picked from a common run-queue by
> the first scheduler that becomes available.
>
> Performance and scalability
> ------------------------------------
>
> - The SMP VM with only one scheduler is slightly slower than the non
> SMP VM. The SMP VM need to to use all the locks inside but as long as
> there are no lock-conflicts the overhead caused by locking is not
> significant (it is the lock conflicts that takes time). This explains
> why it in some cases can be more efficient to run several SMP VM's
> with one scheduler each
> instead on one SMP VM with several schedulers. Of course the running
> of several VM's require that the application can run
> in many parallel tasks which has no or very little communication with
> each other.
>
> - If a program scale well with the SMP VM over many cores depends very
> much on the characteristics of the program, some programs scale
> linearly up to 8 and even 16 cores while other programs barely scale
> at all even on 2 cores.
> This might sound bad, but in practice many real programs scale well on
> the number of cores that are common on the
> market today, see below.
>
> - Real telecoms products supporting a massive number if simultaneously
> ongoing "calls" represented as one or several
> Erlang processes per core have shown very good scalability on dual and
> quad core processors. Note, that these products
> was written in the normal Erlang style long before the SMP VM and
> multi core processors where available and they
> could benefit from the Erlang SMP VM without changes and even without
> need to recompile the code.
>
> SMP performance is continually improved
> ------------------------------------------------------
>
> The SMP implementation is continually improved in order to get better
> performance and scalability. In each service release
> R12B-1, 2, 3, 4, 5 , ..., R13B etc. you will find new optimizations.
>
> Some known bottlenecks
> ---------------------------------
>
> - The single common run-queue will become a dominant bottleneck when
> the number of CPU's or Cores increase.
> Will be visible from 4 cores and upwards, but 4 cores will probably
> still give ok performance for many applications.
> We are working on a solution with one run-queue per scheduler as the
> most important improvement right now.
>
> - Ets tables involves locking. Before R12B-4 there was 2 locks
> involved in every access to an ets-table, but
> in R12B-4 the locking of the meta-table is optimized to reduce the
> conflicts significantly (as mentioned earlier it is the conflicts that
> are expensive).
> If many Erlang processes access the same table there will be a lot of
> lock conflicts causing bad performance especially if these processes
> spend a majority of their work accessing ets-tables.
> The locking is on table-level not on record level.
> Note! that this will have impact on Mnesia as well since Mnesia is a
> heavy user of ets-tables.
>
> ...
>
> Our strategy with SMP
> -----------------------------
>
> Already from the beginning when we started implementation of the SMP
> VM we decided on the strategy:
> "First make it work, then measure, then optimize".
> We are still following this strategy consistently since the first
> stable working SMP VM that we released in May 2006 (R11B).
>
> There are more known things to improve and we address them one by one
> taking the one we think gives most
> performance per implementation effort first and so on.
>
> We are putting most focus on getting consistent better scaling on many
> cores (more than 4).
>
> Best in class
> -----------------
>
> Even if there are a number of known bottlenecks
> the SMP system already has good overall performance and scalability
> and I believe we are best in class
> when it comes to letting the programmer utilize multi -core machines
> in an easy productive way.
>
> /Kenneth Erlang/OTP team, Ericsson
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20080916/acc14868/attachment.htm>