arithmetic anomaly?

[Per Hedeland]

chas@REDACTED wrote:
>
>with the erlang shell i get this:
>
>(1024*1024-1) / 1024.
>1024.00

Which is correctly rounded to 2 decimals (or rather 6 significant
digits) if the "real" value is 1023.9990234375 or thereabouts.

>the following gives a more expected result in erlang, so i wonder
>whether it's not a rounding error somewhere:

It's just a matter of the shell's default output format for floats - see
the description of the io:format() function in the io(3) man page:

1> F = (1024*1024-1) / 1024.
1024.00
2> io:format("~w~n",[F]).   
1024.00
ok
3> io:format("~g~n",[F]).
1024.00
ok
4> io:format("~f~n",[F]).   
1023.999023
ok
5> io:format("~.10g~n",[F]).
1023.999023
ok
6> io:format("~.10f~n",[F]).
1023.9990234375
ok
7>

--Per