Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 148

patients, to generate the multiple repeated observations. The dimensions of
the matrix to be analyzed are 50 by 6. The choice of patients and judges is
nontrivial because the results of the study cannot be generalized beyond
the characteristics of populations from which these individuals are selected.
To increase the generalizability, the investigator selects 50 patients from a
citywide network of hospitals and six expert clinician judges from across
the country. Conducting the study requires the resource to generate its
advice for all 50 patients, and for each of the judges to review and rate
the advice for all of the patients. The reliability of the ratings is estimated
from the resulting objects-by-observations matrix to be 0.82. Using the
Spearman-Brown formula, it is predicted that four judges will exhibit a
reliability of 0.75. Given the time and effort required for the demonstration
study to follow, the investigator decides to use only four judges in the
demonstration study.
Steps to Improving Measurement
Two basic courses of action exist for an investigator if a measurement study
reveals suboptimal reliability or validity: (1) modify the number of inde-
pendent observations in the measurement process (typically affects relia-
bility only); or (2) modify in more substantive ways the mechanics of the
measurement (typically affects both reliability and validity). When a mea-
surement study reveals a low reliability (typically a coefficient of less than
0.70), the investigator can improve it by increasing the number of inde-
pendent observations drawn from the same population. Had the estimated
reliability been too low in our example, the investigator could have added
more judges chosen from the same national group, but it would have come
at a cost. Increasing the number of observations increases the work involved
in conducting each measurement, increasing the time and expense incurred
when conducting the study—and often creating logistical challenges. In
some other situations, as we saw above, a measurement study can yield
higher-than-needed reliability, and the results of the measurement study
can lead the investigator to streamline the study by reducing the number
of independent observations per object. Increasing or decreasing the
number of observations, so long as they represent the same population , can
be assumed to affect reliability only.
Alternatively, the investigator can try to improve the mechanics or instru-
mentation of the measurement in pursuit of increasing the reliability. In our
example, he or she might try better training of the judges, replacing a judge
whose ratings seem unrelated to the ratings of his or her colleagues, or
giving the judges an improved form on which to record the ratings. This
approach, which addresses the substance of the measurement process, often
has greater impact on reliability than merely increasing the number of
observations. This approach also can increase reliability without increasing