Same Problem, Different Resolution

In shopfloor measurements, don’t count partial increments, but do keep a finer-resolution instrument handy.

In a shopfloor measurement of a machined part, the combined variation of the measurement instrument and the operator performing the measurement can leave little room for error—even when the tolerance of the feature is not extremely tight. A part measuring anywhere near the tolerance limit might be rejected as non-conforming. Though a later inspection system may determine the part to be acceptable, sending too many rejected parts on for this extra inspection is wasteful.

To solve this problem, someone usually suggests changing to a finer-resolution gage. After all, many rejected parts measure within just one increment of the tolerance limit. A finer resolution should lessen the frequency of these borderline parts.

But making that change may introduce a new problem. The shop that replaces a 0.01-mm indicator with a 0.001-mm indicator is asking its operators to master the gage more often. While a 0.01-mm indicator might never seem to lose its zero, a 0.001-mm indicator might seem unable to hold its zero. And keep in mind that the 0.01-mm indicator was always adequate outside the borderline region.

Some operators keep the old gage, but try to interpret a finer resolution. That is, they count “partial increments.” This approach is even less acceptable, because a fundamental rule of metrology states that an instrument can never perform better than its resolution. If the resolution of a dial indicator is 0.01 and we measure the part to be +0.05, then the actual value could be +0.04 or +0.06. We should never use this indicator to try to declare a value such as +0.045.

The true solution is compromise. Keep both indicators ready. Use the 0.01 mm indicator for all measurements. Then, any time a part measures within ±0.01 mm (one increment) from the tolerance limit, consider that part “too close to call” using that indicator. Confirm the master with the finer indicator, and then use this indicator to make the final judgment.

About the author: Richard Clark is a metrologist and measurement engineer in Portland, Indiana. His e-mail address is


  • 3D Scanning: Reproducing One-Of-A-Kind Prototypes

    A laser scanning system helps this shop capture the free-form surfaces on a hand-sculpted original. The resulting digitized models are the basis for CAM applications such as programming a CNC machining center. 

  • Composites Machining for the F-35

    Lockheed Martin’s precision machining of composite skin sections for the F-35 provides part of the reason why this plane saves money for U.S. taxpayers. That machining makes the plane compelling in ways that have led other countries to take up some of the cost. Here is a look at a high-value, highly engineered machining process for the Joint Strike Fighter aircraft.

  • Measuring Part Geometry On The Shop Floor

    Measuring workpiece dimensions is relatively simple for machine operators but measuring workpiece geometry which involves more complex comparisons of part shape to an ideal shape--is now also practical on the shop floor. The gaging equipment for doing this is coming down in price while becoming easier to use.

Related Content

There’s a Micrometer for That

Match your micrometer choice to your specific machine shop needs.