[erlang-questions] Any way to correct the round off errors?
Richard Kelsall
r.kelsall@REDACTED
Sun Sep 20 18:08:40 CEST 2009
Witold Baryluk wrote:
>> so we probably shouldn't trust more than 12 digits of precision because
>> each calculation will lose some precision from the end of the number.
> Substraction (and addition) can lose any number of digits you wish.
>
I would be horrified if I added two doubles
0.111111111111 +
0.111111111111
and got
0.225745048327
I have no idea what the IEEE standard specifies, but I can't imagine
anybody ever implementing or using a version that gave this answer.
I would expect at least twelve significant digits of precision.
0.2222222222228765767
But if I added these two doubles
0.111111111111 +
0.000000000000111111111111
I wouldn't be surprised to get
0.111111111111343786587698
which gives the right answer to 12 significant digits, but loses all
of the significant digits in the second number.The same for subtraction.
Richard.
More information about the erlang-questions
mailing list