How to Check Measures for Adequacy - Text Mining Techniques for Healthcare Provider Quality Determination - page 387

of validity covers two large domains. The first has to do with the accuracy of the data and the precision
of the measures that are constructed with these data. The second has to do with the justifiability of the
inferences that are drawn from the data and the measurements.

A search of Medline using the keywords “risk adjustment” and “validation” returned a total of 3 ar-
ticles. It is not yet a concept that is given much consideration, which is why we have so many different
measures of patient severity that can give very different results. However, as providers are more likely
to be rewarded or penalized based upon the results of these measures, the measures themselves will
become more heavily scrutinized.

Background

We first look at the three papers that were found by using the key words “validation” and “risk adjust-
ment”. A recent paper compared the results of two models with a third, internally developed model to
define validation. (Kunadian et al., 2008) Since the models are defined by comparing model values to
each other, this does not give validation as much as it gives reliability. An earlier paper defines validation
by getting similar results on new datasets. Again, this is reliability rather than validation.(Moscucci et
al., 1999) A third paper used the fact that a logistic model predicted accurately, but as discussed in detail
in Chapter 3, accuracy does not imply that the model is adequate or valid.(Mandeep et al., 2003)
True validation takes place when the measurement actually measures what it is supposed to measure.
In other words, a patient severity index must actually measure the actual level of severity of the patient's
condition. A more formal definition of validity is provided at http://www.socialresearchmethods.net/
tutorial/Colosi/lcolosi2.htm. Validity is the best available approximation to the truth or falsity of a given
inference, proposition or conclusion. Consideration of validity attempts to answer the question, is the
severity index true? There are four major types of validity to consider:

1. Convergent validity examines whether there is a relationship between the program and the ob-

served outcome. Or, in our example, is there a connection between the patient severity index and
the patient's level of sickness?

2. Internal validity asks if there is a relationship between the measure and the outcome and whether

the relationship is causal. For example, did the patient's severity level cause the outcomes in mor-
tality, length of stay, and costs?

3. Construct validity asks if there is a relationship between how the concepts are operationalized in

the study to the actual causal relationship. Or in our example, did the measure of severity reflect
the construct of severity, and did the measured outcome reflect the construct of severity? Overall,
we are trying to generalize our conceptualized measure and outcomes to broader constructs of the
same concepts.

4. External validity refers to our ability to generalize the results of our study to other settings. In our

example, could we generalize our results defined using certain providers, to all providers?

The different types of validity focus on the question of whether there is a relationship between the
measure of the outcome and the outcome itself. We must consider first whether a predictive model that
measures the expected mortality is a good measure of the actual mortality. As long as the r 2 value is