In recent years the use of a special Bayesian approach on averaging ‘round-robin’ residual stress data has been implemented. This averaging approach is useful in that it copes with the situation where systematic errors have occurred in one or more of the measurements and thus diminishes the influence of these particular ‘wrong value’ outlier data points. The analyses not only take into account the measurand value, but also the uncertainties associated with each measurand. It should deal with data that may contain individual members with uncertainties larger than the stated error and assumes that the quoted error bar is only a lower bound on the uncertainty. This work shows what could happen when there is a ‘strong mismatch’ in uncertainties when averaging over a limited amount of data. It has been observed that in a case where there are few data points (for example 5 or less), a strong bias can occur towards data points with a relatively small quoted uncertainty compared to other data points with larger quoted uncertainties. A ‘mismatch’ in uncertainty quotation can arise when averaging very good data with poorer data or when averaging with data obtained from other measurement techniques. This effect is demonstrated in this work by using fictitious data and also based on the example of real measurement data obtained by neutron diffraction.