• chiliedogg@lemmy.world
    link
    fedilink
    arrow-up
    1
    arrow-down
    3
    ·
    edit-2
    11 months ago

    It’s not that precision can’t be arbitrarily recorded higher in fraction, it’s that precision can’t be recorded precisely. Decimal is essentially fractional that’s written differently and ignoring every fraction that isn’t a power of 10.

    How can a measurement 3/4 that’s precise to 1/4 unit be recorded in decimal using significant figures? The most-correct answer would be 1. “0.8” or “0.75” suggest a precision of 1/10th and 1/100th, respectively, and sig figs are all about eliminating spurious precision.

    If you have 2 measurement devices, and one is 5 times more precise than the other, decimal doesn’t show it because it can only increase precision by powers of 10.

    In the case of 1/64th above, if you just divide it out it shows a false precision of 1/100,000.