Statistical Methods - Text Mining Techniques for Healthcare Provider Quality Determination - page 111

Since the datasets used to define severity indices are generally too large for a p-value to have mean-
ing, predictive modeling uses other measures of model fit. Generally, too, there are enough observations
so that the data can be partitioned into two or more datasets. The first subset is used to define (or train)
the model. The second subset can be used in an iterative process to improve the model. The third subset
is used to test the model for accuracy. It is also known as a holdout sample.
The definition of “best” model needs to be considered in this context as well. Just what do we mean
by “best”? In a regression model, the “best” model is one that satisfies the criterion of uniform minimum
variance unbiased estimator. In other words, it is only “best” in the class of unbiased estimators. As soon
as the class of estimators is expanded, “best” no longer exists, and we must define the criteria that we
will use to determine a “best” fit. There are several criteria to consider. For a binary outcome variable,
we can use the misclassification rate. However, especially in medicine, misclassification can have dif-
ferent costs. For example, a false positive error is not as costly as a false negative error if the outcome
involves the diagnosis of a terminal disease.
Another difference when using predictive modeling is that many different models can be used, and
compared to find the one that is the best. We can use the traditional regression, but also decision trees
and neural network analysis. We can combine different models to define a new model. Generally, use of
multiple models has been frowned upon because it is possible to “shop” for one that is effective. Indeed,
the nearest neighbor discriminant analysis can always find a model that predicts correctly 100% of the
time when defining the model, but predicts 0% of the time for any subsequent data. When using multiple
models, it is essential to define a holdout sample that can be used to test the results.
Background
Predictive modeling routinely makes use of a holdout sample to test the accuracy of the results. Figure
1 demonstrates predictive modeling. In SAS, there are two different regression models, three different
neural network models, and two decision tree models. There is also a memory based reasoning model,
otherwise known as nearest neighbor discriminant analysis. These models are discussed in detail in
Cerrito (2007). It is not our intent here to provide an introductory text on neural networks; instead, we
will demonstrate how they can be used effectively to investigate the outcome data.
Figure 1 shows that many different models can be used. Once defined, the models are compared and
the optimal model chosen based upon pre-selected criteria. The node labeled Model Comparison is used
for this purpose. It compares all of the models and then chooses the optimal one based upon the pre-
selected criterion. Model comparison can use several different statistics for comparison. The default is
the misclassification rate on the holdout sample. However, if a false negative that results in an ill patient
not getting treatment is more costly to the patient compared to a false positive where a healthy patient
gets unnecessary treatment (or conversely), the model can be optimized based upon a minimization of
costs. It is up to you to choose which measure you want to use to compare models.
Then, additional data can be scored (using the score node as shown in Figure 1) so that new patients
who are admitted subsequently can have a severity level assigned to them. This figture also includes a
data partition so that a holdout sample can be extracted in order to test the model results. It is important
to be able to use the model to score subsequent data. When a patient severity model is defined, it should
be tested on new data to demonstrate reliability.