[erlang-questions] Any way to correct the round off errors?

Richard Kelsall r.kelsall@REDACTED
Sun Sep 20 18:08:40 CEST 2009


Witold Baryluk wrote:
>> so we probably shouldn't trust more than 12 digits of precision because
>> each calculation will lose some precision from the end of the number.
> Substraction (and addition) can lose  any number of digits you wish.
> 
I would be horrified if I added two doubles

0.111111111111 +
0.111111111111

and got

0.225745048327

I have no idea what the IEEE standard specifies, but I can't imagine
anybody ever implementing or using a version that gave this answer.
I would expect at least twelve significant digits of precision.

0.2222222222228765767

But if I added these two doubles

0.111111111111 +
0.000000000000111111111111

I wouldn't be surprised to get

0.111111111111343786587698

which gives the right answer to 12 significant digits, but loses all
of the significant digits in the second number.The same for subtraction.


Richard.


More information about the erlang-questions mailing list