[time-nuts] Standards for units
Tom Van Baak
tvb at leapsecond.com
Tue Apr 3 00:57:37 EDT 2007
> >1/meter/39.37 inches = 0.025400051 meters/inch ...
> The interesting thing is that this result is off by 2.000004000 ppm.
> That's a rather precise error!
> The calculator I used is here:
There are two ways to compute relative error, depending on
which one you call the reference.
We start with 39.37 * 25.4 = 999.998 (exactly) and note that
it is 0.002 (exactly) away from 1000.
You can compute the relative error two ways:
1) 0.002 / 1000.0 = 2 ppm (exactly), or
2) 0.002 / 999.998 = your 2.000004000 ppm number.
I'd guess convention is to use the first number, not the second.
For extra credit, note your number is actually this cool-looking
fraction* whose pattern you will immediately recognize.
* calculated with: echo "80k .002 999.998 / 10 6 ^ * p" | dc
More information about the time-nuts