[erlang-questions] round(1.0e23) ≠ 10²³

[Tony Rogvall]

To help? understand how float works here is a code fragment implementing trunc in erlang:
Here you can see both the representation and how to handle the exponent (for 64 bit IEEE floats).

-module(float).
-compile(export_all).

trunc(X) when is_float(X) ->
    <<S:1,E:11,F:52>> = <<X:64/float>>,
    Exp = E - 1023,
    Sign = if S =:= 0 -> 1; true -> -1 end,
    if Exp >= 0 ->
	    if Exp >= 52 ->
		    Sign*(((1 bsl 52)+F) bsr (52-Exp));
	       true ->
		    Sign*(((1 bsl 52)+F) bsl (Exp-52))
	    end;
       true ->
	    0
    end.

Some examples:

> float:trunc(1.0e22). 
10000000000000000000000

> float:trunc(1.0e23).
99999999999999991611392

> float:trunc(0.9999999999999999). 
0

> float:trunc(0.99999999999999999).
1
...
> float:trunc(float(1 bsl 127)).    
170141183460469231731687303715884105728

> (1 bsl 127).
170141183460469231731687303715884105728

/Tony


On 6 apr 2013, at 23:00, Erik Søe Sørensen <eriksoe@REDACTED> wrote:

> Hi there,
> 
> There are certain common misconceptions around floating-point numbers.
> The key thing to keep in mind is that a) they have limited precision, as they are represented in a fixed number of bits, and b) they are represented in base 2, so numbers which look nice in base 10 and "feel" as if they should not suffer from precision loss, may surprise by doing so.
> 
> In your case, notice that for 10²³ to be represented accurately takes >3*23 = 69 bits -- and Erlang floats are represented in 64 bits (IEEE 754 double precision / C's "double") -- which means that even if all 64 bits were used for the mantissa, the number could not be represented accurately. In fact, there are only 52+1 bits for the mantissa, so the problem starts already at around 10^16:
>   round(10000000000000001.0) => 10000000000000000.
> 
> 
> Hoping this helps,
> /Erik
> 
> 
> 2013/4/6 Pierre Fenoll <pierrefenoll@REDACTED>
> Hi,
> 
> I have notice that on both my R16B 64bit machines (Archlinux & OSX),
> round(1.0e23) = 99999999999999991611392
> Instead of 10²³ as it should.
> 
> I think it's a matter of float to int.
> Do you guys have any idea? Am I doing something wrong?
> 
> Cheers,
> Pierre
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions
> 
> 
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130407/92126972/attachment.htm>