• merc@sh.itjust.works
    link
    fedilink
    arrow-up
    4
    arrow-down
    1
    ·
    1 year ago

    The difference is that when you input a specific, precise floating point number, the number that’s stored isn’t what you entered.

    When you enter integers and store them in ints, as long as the number is small enough, what’s stored is exactly what you entered.

    If you tell your program that the radius of the circle is 0.2 units exactly, it says OK and stores 0.200000000000000011102230246251565404236316680908203125.

    Of course everybody knows that there’s a limit to how many digits get stored. If you tried to store Pi, there’s obviously some point where it would have to be cut off. But, in life we’re used to cutting things off at a certain power of 10. When we say Pi is 3.14 the numbers after the 4 are all zero. If we choose 3.14159 instead, it’s the numbers after the 9 that are zero. If we represent these as fractions one is 314/100 the other is 314159/100000. The denominator is always a power of 10.

    Since computers are base 2, their denominator is always a power of 2, so there’s a mismatch between the rounded-off numbers we use and the rounded-off numbers the computer uses.