## Exclusive Content & Downloads from ASQ

# The Effect of Ignoring Small Measurement Errors in Precision Instrument Calibration

Summary: This paper discusses the effect of measurement errors in both variables when using the simple linear regression model. It is often stated that if the measurement error in x is small, then we can ignore this error and fit the model to data using ordinary least squares. There is some ambiguity in the statistical literature concerning the exact meaning of a small error. For example, Draper and Smith (1981) state that if the measurement error variance in x is small relative to the variability of the true x's, then "errors in the x's can be effectively ignored." See Montgomery and Peck (1983) for a similar statement. Scheffe (1973) and Mandel (1984) argue for a second criterion, which may be informally summarized that the error in x should be small relative to (the standard deviation of the observed Y about the line)/(slope of the line). We argue that for calibration experiments, both criteria are useful and important; the former for estimation of x given Y, and the latter for the lengths of confidence intervals for x given Y.

### Other Ways to Access content:

### Join ASQ

### Subscribe to *Journal of Quality Technology*

**Topics:**Statistics**Keywords:**Statistics,Calibration,Inverse regression,Least squares,Linear regression,Measurement error**Author:**Carroll, Raymond J.; Spiegelman, Clifford H.**Journal:**Journal of Quality Technology