Measurement Fundamentals - Evaluation Methods in Biomedical Informatics - page 133

teristic of the resource itself but rather of the environment in which the
resource is installed. As shown in Table 5.4, content validity is relatively easy
to assess, but it provides a relatively weak argument, grounded in subjec-
tive judgment, for the validity of a measurement process. Nonetheless, the
content validity of a measurement process cannot be assumed and must be
verified through appropriately designed explorations.
Criterion-Related Validity
For criterion-related validity the central question is different: Do the results
of a measurement process correlate with some external standard or predict
an outcome of particular interest? For example, do those who score highly
on a scale purported to measure computer literacy also learn more quickly
to navigate an unfamiliar piece of software? Does a scale that rates the
quality of radiotherapy treatment plans identify treatment plans associated
with longer patient survival? Determination of criterion-related validity
depends on the identification of specific criteria that will be accepted as rea-
sonably definitive standards and for which reliable and valid measurement
methods already exist. If the measurement process under study is to be con-
sidered valid, the correlation with a criterion measure would be expected
to be moderately high, with coefficients of at least 0.5, and preferably
higher.
Unlike content validity, which can be assessed through inspection of the
measurement instruments and processes themselves, determination of cri-
terion-related validity requires a study where measurements are made, on
a representative sample of objects, using the instrument being validated as
well as the instruments needed to assess the criterion. Using our previous
example, estimating the criterion-related validity of a computer literacy
scale in a formal measurement study requires that the scale be completed
by a sample of health professionals who also try their hand at using
an unfamiliar piece (or pieces) of software. Their scores on the literacy
scale would then be analyzed in relation to the time taken to master
the unfamiliar software. The greater the statistical correlation between
literacy scores and mastery time, the greater the criterion-related validity
of the scale. Note that in this case, the polarity of the relationship would
be expected to be negative: lower mastery times associated with increased
literacy scores.
When the criterion is an attribute that can only be known in the future,
criterion-related validity is often referred to as “predictive” validity. It is
often possible to identify predictive standards for validation of new mea-
surement methods, but predictive validation studies can take months or
years to complete. Using survival rates of patients as a criterion to validate
methods for rating the quality of radiotherapy treatment plans is an
example of a predictive validity study. When the criterion is an attribute
that can be measured at the same time as the measurement process under