[erlang-bugs] [erlang-questions] Erlang on Xeon Phi

[Mikael Pettersson]

Jasiek Stypka writes:
 > Hello everyone,
 > 
 > Does anybody have any experience with running Erlang programs on Intel
 > Xeon Phi coprocessor? I would like to carry out some experiments on this
 > platform concerning multicore computing, however I am struggling to run
 > Erlang VM on that architecture.
 > 
 > It seems to be binary incompatible with x86_64 architectures. When I try
 > to run the normal Erlang VM v17.3 on it, I get:
 > /home/me/lib/erlang/bin/erl: line 28: /home/me/lib/erlang/erts-6.1/bin/erlexec: cannot execute binary file
 > /home/me/lib/erlang/bin/erl: line 28: /home/me/lib/erlang/erts-6.1/bin/erlexec: Success
 > 
 > So I figured out that I need to use cross compilation to compile the VM
 > specially for this architecture. After some reading and numerous trials
 > and errors, I ended up compiling the EVM with these commands:
 > 
 > $ ./configure --host=k1om-unknown-linux-gnu --build=x86_64-pc-linux-gnu --without-termcap --prefix=$HOME/lib/my_erlang_compilation-17.3 CC=icc CFLAGS=-mmic
 > $ make
 > 
 > The first command somehow finishes without any errors, but when running
 > `make` I finally get:
 > 
 > /tmp/iccvaLP3vas_.s: Assembler messages:
 > /tmp/iccvaLP3vas_.s:25794: Error: `mfence' is not supported on `k1om'
 > /tmp/iccvaLP3vas_.s:25800: Error: `mfence' is not supported on `k1om'
 > /tmp/iccvaLP3vas_.s:26388: Error: `mfence' is not supported on `k1om'
 > /tmp/iccvaLP3vas_.s:26394: Error: `mfence' is not supported on `k1om'
 > make[3]: *** [obj/k1om-unknown-linux-gnu/opt/smp/erl_alloc_util.o] Error 1
 > 
 > This seem to be connected with memory barrier instructions and looks like
 > it's incompatible. Does this mean that the Erlang VM cannot be run on
 > this architecture?
 > 
 > Thanks in advance for any suggestions,

I ported Erlang/OTP R15B02 to the Xeon Phi in 2012, using Intel's port
of gcc-4.7.0 for the Xeon Phi.  As you've noticed, the Xeon Phi is not a
"normal" x86_64: in fact it's a P5-era core upgraded with basic 64-bit
support and a new vector unit, but it lacks SSE2 and related features that
the rest of the world consider part of "x86_64".

Anyway, the only patch I had to apply is the following:

--- otp_src_R15B02/erts/include/internal/i386/ethr_membar.h.~1~	2012-09-03 11:58:05.000000000 +0200
+++ otp_src_R15B02/erts/include/internal/i386/ethr_membar.h	2012-10-22 13:56:54.000000000 +0200
@@ -48,12 +48,12 @@ ethr_cfence__(void)
 static __inline__ void
 ethr_mfence__(void)
 {
-#if ETHR_SIZEOF_PTR == 4
+#if ETHR_SIZEOF_PTR == 4 || !defined(__SSE2__)
     if (ETHR_X86_RUNTIME_CONF_HAVE_NO_SSE2__)
 	ETHR_NO_SSE2_MEMORY_BARRIER__;
     else
 #endif
-	__asm__ __volatile__ ("mfence\n\t" : : : "memory");
+	__asm__ __volatile__ (".byte 0x0f,0xae,0xf0 #mfence\n\t" : : : "memory");
 }
 
 static __inline__ void

The port was just an experiment as I was working a lot with Xeon Phi
during 2012/2013.  61 cores / 244 threads, lots of memory and cache
bandwidth, Linux kernel, lame userspace, small RAM (4 or 8 GB I think),
very slow sequential integer performance.  Hopefully newer models are better.