Developing and Improving Measurement Methods - Evaluation Methods in Biomedical Informatics - page 153

matics studies. Here we consider general strategies that may be followed
from a diagnostic process using part-whole correlations. For brevity's
sake, we refer to corrected part-whole correlations simply as part-whole
correlations:
1. If all part-whole correlations are reasonably large but the reliability is
too low: Add equivalent observations to the measurement process.
2. If many part-whole correlations are low: Something affecting all obser-
vations is fundamentally amiss. There are likely to be only small differences
in the scores among objects. Check aspects of the measurement process that
relate to all observations; for example, if human judges are using a rating
form, the items on the form may be phrased misleadingly.
3. If one (or perhaps two) observations display low part-whole correla-
tions: First try deleting the misbehaving observation(s). The reliability may
be higher and the entire measurement process more efficient if so pruned.
Alternatively, try modifying or replacing the misbehaving observation(s),
but always keep in mind that selectively deleting observations can affect
what is being measured.
4. If two or more observations display modest part-whole correlations
while the others are high: This situation is ambiguous and may indicate that
the observations as a group are measuring two or more different attributes.
In this case, each subset displays high intercorrelation of its member obser-
vations, but the observations from different subsets are not correlated with
each other. This possibility cannot be fully explored using part-whole cor-
relations and requires either careful inspection of the full intercorrelation
matrix or use of more advanced statistical techniques, such as principal com-
ponent or factor analysis. 2,3 If the investigator expected the observations to
address a single attribute and in fact they address multiple discrete attrib-
utes, the entire measurement process is not performing as intended and
should be redesigned. There is no evidence in this situation that the
attribute hypothesized to be measured exists as such. (To play out an
example in detail, complete Self-Test 6.2, below.)
If a specific observation is not well behaved (see Outcome 3 above),
several things may be happening, and it will be necessary to pinpoint the
problem in order to fix it. For example, consider items on a questionnaire
as a set of observations. A misbehaving item may be so poorly phrased that
it is not assessing anything at all, or perhaps the particular objects—in this
case, questionnaire respondents used for the measurement study—lack
some specific knowledge that enables them to respond to the item. In this
case an improved part-whole correlation may be observed if the item is
tested on a different sample of objects. Alternatively, the item may be well
phrased but, on logical grounds, does not belong with the other items on
the scale. This situation can be determined by inspecting the content of the
item, or, if possible, talking to the individuals who completed it to see how
it was interpreted. Because it is usually necessary to collect new measure-