Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 183

3. Content validity is somewhat built into the study through the choice
of expert raters. It could be further ensured by asking the judges to write
the rationale for their ratings for a subset of the patients and checking these
rationales against published standards of care where possible. Criterion-
related validity could be explored by selecting a subset of patients for whom
the system's advice had been followed and comparing clinical outcomes
seen in patients for whom the advice had been highly rated in comparison
with those patients for whom the advice was poorly rated. Construct valid-
ity might be assessed by examining some case properties with which the
system's correctness might be hypothesized to be correlated. For example,
cases that are more complex, as measured by the number of clinical vari-
ables they include, might be hypothesized to be truly more difficult and thus
expected to generate lower scores from the raters. Cases with diagnoses that
are more prevalent might be hypothesized to be less difficult and expected
to generate higher scores.

Self-Test 6.2
1. More observations are needed to increase the reliability. The obser-
vations in the set are generally well behaved.
2. It appears that two attributes are being measured. Items 1 to 3 are
measuring one attribute, and items 4 to 6 are measuring the other.
3. Judge H displays the highest corrected part-whole correlation (0.55)
and thus can be considered the “best” judge. Judge E is a close second with
a part-whole correlation of 0.50. Judge C may be considered the worst
judge, with a part-whole correlation of 0.27. Removing judge C raises the

reliability from 0.29 to 0.54 in this example. Such a large change in relia-
bility is seen in part because the number of objects in this example is small.
Judges B and D can in some sense be considered the worst, as they ren-
dered the same result for every object and their part-whole correlations
cannot be calculated.

-

Self-Test 6.3
The object class comprises patients who are to be admitted. In this case,
“observers” (five levels), “times of day” (three levels), and “hospitals” (four
levels) are facets of the measurement study.

Self-Test 6.4
The “observer” facet belongs to the “judges” category. The “times of day”
and “hospitals” facets belong to the “logistical factors” category.