[erlang-bugs] Rounding errors with hipe

[Mikael Pettersson]

Fredrik Svahn writes:
 > I get rounding errors for some odd cases when running one of the programs in
 > the Computer Language Shootout with Hipe. So far I have only seen it on
 > 32-bit (x86) targets.
 > 
 > I am guessing this might be a known problem/limitation in Hipe. I have
 > however not seen any previous bug reports on it, so better safe than sorry;
 > Here is a minimal test program with different results for byte code and
 > natively compiled code.
 > 
 > > otp_src_R12B-5/bin/erlc testmath.erl
 > > otp_src_R12B-5/bin/erl -noshell -run testmath main
 > 0.0
 > > otp_src_R12B-5/bin/erlc +native testmath.erl
 > > otp_src_R12B-5/bin/erl -noshell -run testmath main
 > 3.122502256758253e-17
 > 
 > > uname -a
 > Linux host 2.6.16.53-0.16-smp #1 SMP Tue Oct 2 16:57:49 UTC 2007 i686 i686
 > i386 GNU/Linux

First of all you do know that floating-point (FP) is inherently
imprecise, right?

However, the problem here is the quirky nature of the old x87 FP unit,
which is what 32-bit x86s use by default in most OS environments.
(I believe MacOSX is one of the few to default to SSE2 even in 32-bit.)

The x87 computes internally with an 80-bit format different from
standard IEEE 64-bit floats. It does however round the result of
each computation according to the precision control (which defaults
to the 80-bit extended format), and it rounds when storing values
to IEEE 64-bit floats in memory. So the final result is highly
sensitive to the sequence of loads, arithmetics, and stores done
during the computation.

The BEAM virtual machine will store each intermediate result to
memory, causing rounding to the IEEE 64-bit format.

HiPE on x86-32/x87 will keep intermediate results in the x87
register file (which technically is a stack) as long as possible.
The benefit is improved performance from avoiding memory accesses.
It also improves accuracy by avoiding some rounding operations.

And that's the problem. The nature of FP is such that computing
a more accurate result can be considered an error.

A possible solution might be to program the x87's rounding control
to always round to standard double precision. That should make BEAM
and HiPE produce similar results, but it might break FP library
routines.

Having HiPE store each intermediate value just for the sake of
BEAM compatibility would be awful.

The real solution is to forget about x87 and go to x86-64/SSE2.

/Mikael