We are having issues with stray decimal values anbd rounding. I research the forum and did not find anything except function use.

Can someone explain why expression masking only works some of the time?

Example:
MAP1 DLA$,S,16," ############.##"
MAP1 MASK16$,S,16,"#####.##########"
MAP1 TEMP,F,8
MAP1 NEWTEMP,F,8,0
TEMP = 140.58000000007
NEWTEMP=NEWTEMP+TEMP USING DLA$
PRINT USING MASK16$,NEWTEMP
END

Results: 140.5800000001

I tested this under unix and windows with same result. However, under the Alphamicro the mask was applied as I expected to two decimal places with result of 140.5800000000

I have used this method since the early 80's, but now wonder why AShell treats this differently. I know it has something to do with precision but why does the mask not apply?

In addition if I add the following code (multiplication) the mask applies:

NEWTEMP=NEWTEMP*100 USING DLA$
PRINT USING MASK16$,NEWTEMP

Results: 14058.0000000000

This problem, while interesting, has frustrated many people in the migration from the Motorola 68K architecture (with its 48 bit internal floating point math) to the standard 64 bit internal floating point environment of the various operating systems where A-Shell runs.

The basic problem is that contrary to intuition, the increased precision of the 64 bit representation actually increases the rounding errors (when trying to represent decimal fractions in binary exponential format) rather than decreasing them.

So something simple like adding a column of fractional numbers, even if there are only a few digits past the decimal point, will give you a different raw result with 64 bit floating point arithmetic vs 48 bit.

Over the course of many years and many applications, we came up with a series of refinements/adjustments to, for the most part, force the 64 bit arithmetic to emulate 48 bit so that A-Shell can generate the same results as AMOS. (The main trick for this involves rounding arithmetic results to 48 bit representation before using them as input to subsequent operations. Another trick, which is optionally set via SET FPROUND, involves adding a fudge factor several decimal points to the right of the last significant digit in a format USING mask.)

But neither of those can deal with the case where your mask (or the requested precision) is actually requiring for so many significant digits.

To illustrate the problem, we can just look at the impossibility of representing a relatively simple number, 140.58, in floating point representation:

What that shows is that while the F6 and F8 formats end up with the same representation (which is a side effect of the fact they share an internal expression handling path), they're both off slightly starting in the 14th significant digit). And the F4 representation is off in the 9th significant digit.

Note that we haven't even done any arithmetic. The fact is that there is no perfect way to represent the decimal value 140.58 as a binary mantissa and exponent. The only thing we can do is manage rounding in order to achieve a consistent level of precision within the limits of the hardware and representation scheme.

When printed with 11 significant digits, they all look perfect because of rounding. But at 16 digits, they're all off. And using a mask with 16 significant digits effectively forces 16 digit significance.

In your example, it isn't even necessary to start with the slightly-off value of 140.58000000007; as my example shows, the closest we can come to representing 140.58 in floating point format is approximately 140.58000000007 anyway.

The purpose of using the mask for rounding (as opposed to some purely mathematical technique) is to eliminate those straggling digits way off to the right. But that requires that the mask have fewer positions to the right of the decimal. The mask you are using may be giving you the benefit of rounding errors in the 16th digit down to 15 digits, but that isn't sufficient to handle the problem since the precision breakdown is occurring in the 14th significant digit.

As to why this worked differently under AMOS, I'm not quite sure. It appears that maybe they are internally limiting precision to less than 15 digits prior to applying a mask. I tried the following with BasicPLUS on AMOS 8.2 ...


(Apparently the AMOS BasicPLUS doesn't support significant 16, so I changed it to 15.) The same exact program under A-Shell:

So we get the exact same raw values for F8 and F4, but oddly AMOS rounded the F6 value to 2 digits even without a mask? And with the mask, they all get rounded at least one digit prior to the end of the mask.

I don't have any explanation for the unexpected rounding of the raw F6 value (and why it stands out from the F8 and F4 representations), although it probably doesn't matter in your inquiry because you appear to be using F8 anyway.

As for the mask rounding, I would argue that the A-Shell version is correct, since the straggling digits 75 occur in the 11th position, which should cause the 10th position to round up from 0 to 1. Apparently the AMOS version is setting some kind of internal limit on the precision when formatting a mask. Obviously it can't do this with shorter masks, or 140.58007 as "###.####" would appear as "140.5800" instead of the correct "140.5801". I don't know if it is a matter of luck, or some kind of Solomonic decision to truncate after ~14 digits or so, but it seems unexpected, at least to me.

I need to think about whether it makes sense to try to emulate this oddity. Obviously AMOS compatibility is an important goal. But there are many AMOS developers and especially users who never worked on AMOS and for whom a more important goal would be for the language to work as one would expect. Although arguably, "expectations" at this level of floating point precision are probably not well founded in any case, since at this level the behavior is likely to change from one numeric value to the next.

Let me ponder this a bit...

Why does the mask problem arise even when there is no explicit floating point variable?

MAP1 STR1$,S,6,"140.58"
MAP1 MASK,S,14,"###.##########"
PRINT STR1$
PRINT STR1$ USING MASK

140.58
140.5800000001

Is it because STR1$ is converted to float before the mask is applied?

Exactly. STR1$ is converted to a numeric value (i.e. F8 internally) before being processed by the ###.########## (numeric) mask conversion.

I guess the question I have for JR is what the intent of all those trailing digits in the MASK16$ is.

When following the USING DLA$ expression (which rounds to 2 digits after the decimal), the best we can hope for from MASK16$ is a string of 8 trailing zeroes. Maybe that has some aesthetic purpose in the application, but it seems like it would be misleading to the user who might interpret all those trailing zeroes as significant digits (which they wouldn't really be after the rounding operation).

That said, if you have hundreds of examples of this kind of logic scattered throughout the application, I'm not adamantly opposed to implementing some kind off switch to truncate after some number of trailing digits to simulate the mysterious (but fortuitous in this case) AMOS behavior.

Well, now that is a lot to ponder indeed. I used MASK16$ only to see what was actually in the variable after the addition USING DLA$ to see if DLA$ truncated thus rounded to two decimals.

The primary issue we are having at Compupay is these small decimal values end up getting stored in data files (unknown to us). We have diagnostic reports comparing summarized details to calculated quarterly values and sometimes comparing 140.58 to 140.58 says they do not equal. And they don't if the printed values are unmasked showing these tiny decimals. We'll have to perform rounding differently.

By the way. Thanks for your time to demonstrate what is happening. Makes it quite clear.

The complications of storing fixed-decimal values (like dollars and cents) in floating point representation are surprising to most people, given how nearly everyone uses floating point for this purpose.

Assuming that you never had to store values larger than +/- 21 million dollars (2 giga-cents), I4 would be preferable (but that wasn't an option in the early days). B5 extends the range to about +/- 500 giga-cents (5 billion dollars). But it's a rather non-standard format if you're contemplating tapping into the same data files from outside sources. (And not entirely coincidentally, that's about 11.5 significant digits, similar to what F6 offers, except that when you exceed the limit, it wraps around from positive to negative instead of just getting fuzzy.)

Sticking with F8 representation, you just have to accept that any fractional values are at risk of having these tiny "lint" values out in the 14th to 16th digit. You can live with that as long as you always use a formatting mask for presentation and take care not let them accumulate when doing arithmetic. One approach is to just use a rounding mask after each operation, e.g.

That way, the rounding error doesn't grow with each operation.

Another technique is to use SCALE so that all the stored values are effectively integers. That's a nice trick, but unless you started with it, converting to it is a recipe for disaster since there is no warning when you forget to declare the SCALE value in a program, which will result in it reading/writing file values that are out of sync with their intended values by a factor of 100. Plus, it's a constant concern when using subroutines, libraries, etc. that have to meticulously remember to scale everything.

Yet another idea, which may sound weird at first, is to store your values as strings (applying a mask during the assignment). That also makes them easy to access from external tools, since everything can understand string values. But there again, you have to be very careful not to accidentally concatenate rather than add, due to the overloading of the + operator.

Personally, I've mostly relied on the rounding mask method, which is relatively easy to understand and implement, as well as being somewhat self-documenting. (In contrast, scaling and rounding methods involving multiplication and division often seem somewhat opaque when modifying code 10 years later, and you find yourself wondering whether the value you're working with is in cents or dollars, or what.)

Maybe it's time for I,6 and/or I,8? 🙄

Possibly, but that's a subject for this thread:  I,8 and B,8 types?