Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 149

the resources required to conduct the measurement. It is important to
understand, however, that such changes can affect what is being measured
and thus can affect both reliability and validity. When an investigator
responds to a measurement study result by changing the number of obser-
vations, it is typically not necessary to repeat the study because the impact
of the change can be predicted from the Spearman-Brown formula. When
the changes are more fundamental (e.g., a change in the format of a rating
instrument or a change in the population from which judges are selected),
it may be necessary to repeat the measurement study, possibly going
through several iterations until the process reaches the required level of
performance.
Self-Test 6.1
With reference to the example described earlier (see page 147):
1. What is the predicted reliability of this measurement process using one
judge only? Would you consider this figure acceptable?
2. In the measurement study, the ratings were generated on a 1 to 4
response scale and had a mean of 2.3 with a standard deviation of 0.8.
What was the magnitude of the standard error of measurement?
3. How might validity be explored in this hypothetical measurement study?
[Answers are found at the end of the chapter.]
Using Measurement Studies to Diagnose
Measurement Problems
In this section we discuss how the investigator can decide, based on mea-
surement study results, which specific strategies to pursue to improve
measurement.
Analyzing the Objects-by-Observations Matrix
The diagnostic process entails some further analysis of the objects-by-
observations matrix to determine which of the observations, if any, is
eroding the reliability. Recall that each independent observation in the
measurement process is hypothesized to assess the same attribute. If these
observations do assess the same attribute, the results of each pair of obser-
vations across a sample of objects tend to be at least modestly correlated.
That is, an object with a high score on one observation tends to also have
a high score for the other observations. Observations that assess different
attributes tend to be uncorrelated. This matrix of intercorrelations may
be computed directly from the objects-by-observations matrix generated
by the measurement study. Pearson product-moment correlations are