Information Technology Reference
In-Depth Information
METHODS OF VALIDATION
Your choice of an appropriate methodology will depend upon your objec-
tives and the stage of your investigation. Is the purpose of your model to
predict—will there be an epidemic? to extrapolate—what might the
climate have been like on the primitive Earth? or to elicit causal mecha-
nisms—is development accelerating or decelerating? Which factors are
responsible?
Are you still developing the model and selecting variables for inclusion,
or are you in the process of estimating model coefficients?
There are three main approaches to validation:
1. Independent verification (obtained by waiting until the future
arrives or through the use of surrogate variables).
2. Splitting the sample (using one part for calibration, the other for
verification).
3. Resampling (taking repeated samples from the original sample and
refitting the model each time).
Independent Verification
Independent verification is appropriate and preferable whatever the objec-
tives of your model and whether selecting variables for inclusion or esti-
mating model coefficients.
In soil, geologic, and economic studies, researchers often return to the
original setting and take samples from points that have been bypassed on
the original round. See, for example, Tsai et al. [2001].
In other studies, verification of the model's form and the choice of vari-
ables are obtained by attempting to fit the same model in a similar but
distinct context.
For example, having successfully predicted an epidemic at one army
base, one would then wish to see if a similar model might be applied at
a second and third almost but not quite identical base.
Stockton and Meko [1983] reconstructed regional-average precipitation
to
A
.
D
. 1700 in the Great Plains of the United States with multiple linear
regression models calibrated on the period 1933-1977. They validated the
reconstruction by comparing the reconstructed regional percentage-of-
normal precipitation with single-station precipitation for stations with
records extending back as far as the 1870s. Lack of appreciable drop in
correlation between these single station records and the reconstruction
from the calibration period to the earlier segment was taken as evidence
for validation of the reconstructions.
Graumlich [1993] used a response-surface reconstruction method to
reconstruct 1000 years of temperature and precipitation in the Sierra
