Exclusive Content & Downloads from ASQ

The Effect of Ignoring Small Measurement Errors in Precision Instrument Calibration

Summary: This paper discusses the effect of measurement errors in both variables when using the simple linear regression model. It is often stated that if the measurement error in x is small, then we can ignore this error and fit the model to data using ordinary least squares. There is some ambiguity in the statistical literature concerning the exact meaning of a small error. For example, Draper and Smith (1981) state that if the measurement error variance in x is small relative to the variability of the true x's, then "errors in the x's can be effectively ignored." See Montgomery and Peck (1983) for a similar statement. Scheffe (1973) and Mandel (1984) argue for a second criterion, which may be informally summarized that the error in x should be small relative to (the standard deviation of the observed Y about the line)/(slope of the line). We argue that for calibration experiments, both criteria are useful and important; the former for estimation of x given Y, and the latter for the lengths of confidence intervals for x given Y.

Anyone with a subscription, including Site and Enterprise members, can access this article.

Other Ways to Access content:

Join ASQ

Join ASQ as a Full member. Enjoy all the ASQ member benefits including access to many online articles.

Subscribe to Journal of Quality Technology

Access this and ALL OTHER Journal of Quality Technology online articles. You'll also receive the print version by mail.

  • Topics: Statistics
  • Keywords: Statistics,Calibration,Inverse regression,Least squares,Linear regression,Measurement error
  • Author: Carroll, Raymond J.; Spiegelman, Clifford H.
  • Journal: Journal of Quality Technology