Convert double to float goes wrong - C#
The line float BoundsHeight = 0.1f;
converts .1 to the nearest value representable in float
, resulting in BoundsHeight
being 0.100000001490116119384765625
The line double BoundsY = 2,333,638.6551984739;
similarly converts to double
, setting BoundsY
to 2,333,638.6551984739489853382110595703125.
The line float MinY = BoundsY;
converts that to float
, setting MinY
to 2,333,638.75.
The line double MaxY_Defect = BoundsY + BoundsHeight;
computes using double
(I presume; I am not familiar with C# semantics), setting MaxY_Defect
to 2,333,638.7551984754391014575958251953125.
The line float MaxY = (float)(MaxY_Defect);
converts that to float
, setting MaxY
to 2,333,638.75.
Then we can see that MinY
and MaxY
have the same value, so of course MaxY-MinY
is zero.
Quite simply, float
does not have enough precision to distinguish between 2,333,638.6551984739489853382110595703125 and 2,333,638.7551984754391014575958251953125. At the scale of 2,333,638, the distance between adjacent representable numbers in the float
format is .25. This is because the format has 24 bits for the significand (the fraction portion of the floating-point representation). 2,333,638 is between 221 and 222, so the exponent in its floating-point representation scales the significand to have bits representing values from 221 to 2−2 (from 21 to −2, inclusive, is 24 positions). So changing the significand by 1 in its lowest bit changes the represented number by 2−2 = .25.
Thus, when 2,333,638.655… and 2,333,638.755… are converted to float
, they have the same result, 2,333,638.75.
You cannot use float
to distinguish between coordinates or sizes that are this close at that magnitude. You can use double
or you might be able to translate the coordinates to be nearer the origin (so their magnitudes are smaller, putting them in a region where the float
resolution is finer).
As long as the final result is small, you could do the intermediate calculations using double
but still represent the final result well using float
.
For additional information on floating-point arithmetic, I recommend Handbook of Floating-Point Arithmetic by Muller, Brunie, de Dinechin, Jeannerod, Joldes, Lefèvre, Melquiond, Revol, and Torres. A PDF of a prior edition appears to be available here. The official IEEE 754-2019 Standard for Floating-Point Arithmetic is available here.