[erlang-questions] Any way to correct the round off errors?

[Witold Baryluk]

Dnia 2009-09-20, nie o godzinie 17:08 +0100, Richard Kelsall pisze:
> Witold Baryluk wrote:
> >> so we probably shouldn't trust more than 12 digits of precision because
> >> each calculation will lose some precision from the end of the number.
> > Substraction (and addition) can lose  any number of digits you wish.
> > 
> I would be horrified if I added two doubles
> 
> 0.111111111111 +
> 0.111111111111
> 
> and got
> 
> 0.225745048327
> 
> I have no idea what the IEEE standard specifies, but I can't imagine
> anybody ever implementing or using a version that gave this answer.
> I would expect at least twelve significant digits of precision.
> 
> 0.2222222222228765767
> 
> But if I added these two doubles
> 
> 0.111111111111 +
> 0.000000000000111111111111
> 
> I wouldn't be surprised to get
> 
> 0.111111111111343786587698
> 
> which gives the right answer to 12 significant digits, but loses all
> of the significant digits in the second number.The same for subtraction.
> 
> 
> Richard.
Maybe i written not clearly. I was thinking more about such situation:

> 0.111111111111 - 0.111111111113.
-1.9999973899231804e-12
-0.000000000001999997389923180435
>

It have only one correct significant decimal digit.


-- 
Witold Baryluk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: To jest cz??? wiadomo?ci podpisana cyfrowo
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20090920/c79db5b6/attachment.bin>