arithmetic anomaly?
chas@REDACTED
chas@REDACTED
Wed Aug 11 07:05:55 CEST 2004
with the erlang shell i get this:
(1024*1024-1) / 1024.
1024.00
the equivalent expression on the same platform (NetBSD on Intel) in
the clisp implementation of common lisp gives me this:
(/ (- (* 1024 1024) 1) 1024.0)
1023.999
ocaml:
# float_of_int (1024*1024-1) /. 1024.0;;
- : float = 1023.99902344
python:
(1024*1024-1) / 1024.
1023.9990234375
the following gives a more expected result in erlang, so i wonder
whether it's not a rounding error somewhere:
(1024*1024-6) / 1024.
1023.99
More information about the erlang-questions
mailing list