[erlang-questions] Maths Problem -> 2452.45*100. = 245244.99999999997

[Richard A. O'Keefe]

On 28/11/2013, at 7:02 AM, Alexander Alexeev wrote:

> Here is a real world example. We are calculating a commission:
> 
> 28 x 85 x 312 x 5 / 0.85

In VisualAge Smalltalk, open a Workspace, type
	28 * 84 * 312 * 5 / 0.85s2
and then click on the spectacles icon.  The answer
	4368000
is inserted and selected.

0.85s2 is a ScaledDecimal representing an exact decimal fraction
with 2 digits after the decimal point.  On the IBM mainframes,
this was aimed at exploiting z/Series decimal fixed point
instructions, and the representation in VA/ST is 18 bytes.
One byte holds a precision (number of digits).
One byte holds a scale (number of digits after the decimal point).
The remaining 16 bytes hold 2 packed decimal digits per byte,
with one of those "digits" being a sign, so the maximum
precision is 31 and the maximum scale is 255.
> 
> In this case decimal gives you a rounding error +/- 0.01 USD. With
> _any_ chosen precision.

As we just saw, that's not true.  It might be true of decimal floats,
but it's not true of NUMERIC(3,2) arithmetic in this case.
Other Smalltalks do not have the same hard limit on the magnitude of
ScaledDecimal numbers, but they have their own quirks.

There's more than one way to specify decimal arithmetic for a computer
and some of them are better than others.