Assessing Model Adequacy - Environmental Modelling: Finding Simplicity in Complexity - page 422

achieving a good fit to historical observations is itself a
common method for estimating appropriate values of the
input parameters. Model calibration or tuning is a subject
with an extensive literature; see, for example, Kennedy
and O'Hagan (2001) and Rougier (2009). All that we are
looking for at this stage is to be reasonably confident that
the model is sufficiently reliable to merit such a tuning
effort. A simple approach for making such an assessment
is to make many evaluations of the model using a space-
filling design in the input parameters and to determine
which choices of input parameter lead to the best fits to
the field data. For high-dimensional input spaces, it may
not be directly feasible to make evaluations over all areas
of the input space to an acceptable level of concentration.
In such cases, we often use an iterative design, eliminating
all input choices that give very poor fits within the first-
stage design and placing second-stage designs centred on
those evaluations that have givenmore reasonable fits and
continuing in this manner until a collection of relatively
good fits have been found.
This process is sometimes referred to as history match-
ing (see, for example, Craig et al ., 1997). We are not trying
to determine the best choice of input parameters but sim-
ply todetermine if there is some subcollection that gives an
acceptable match to historical data. It might be that every
evaluation that we make of the model provides such a
poor fit to the historical data that we reach the conclusion
that external discrepancy is so large as to render the model
unacceptable for practical use. Otherwise, assessment of
the order of magnitude discrepancy between model and
data in regions of good fit gives us a guide to the mag-
nitude of external discrepancy. This method of tuning is
only likely to give meaningful results if we have access
to a large quantity of field data relative to the number of
parameters that we may vary; otherwise, it is highly likely
that we will overfit themodel to the data. If our assessment
of external variance appears to be negative for many com-
ponents of z , because the differences between f ( x )and z
are small compared to observational plus internal discrep-
ancy errors, then this suggests we have possibly overfitted
the model, and further investigation may be required.
In order to carry out the above analysis, we must
make many evaluations of the model within a reasonable
length of time. For many problems, this is not a realistic
possibility. In such cases, we may employ the method
of model emulation. Emulation refers to the expression
of our beliefs about the function f ( x ) by means of a
fast stochastic representation, which we can use both to
approximate the value of the function over the input
space and also to assess the uncertainty that we have

introduced from using this approximation. For example,
we might represent our beliefs about the i th component
of f ( x ) in the form:

j g i ( x )

f i ( x ) = β ij + u i ( x ) (26.2)

where each g j ( x ) is a known deterministic function of x ,
for example a polynomial term in some sub-collection
of the elements of x ,the

β ij are unknown constants to
estimate and u i ( x ), the residual function, is specified as
having zero mean and constant variance

2
i

σ for each x ,

with a correlation function c i ( x , x ) corr( u i ( x ), u i ( x )),

which only depends on the distance between x and x .
There are many possible choices for the form of the
c i ( x , x ). If we want to carry out a full probabilistic analysis
thenwemay suppose, for example, that u i ( x ) is a Gaussian
process, so that the joint distribution of any subcollection
of values of u i ( x ) for different choices of x is multivariate
normal.
There is an extensive literature on the construction of
emulators for computer models, based on a collection of
model evaluations (see, for example, O'Hagan, 2006, and
MUCM, 2009). Given these evaluations, we may choose
our functional forms g j ( x ) and estimate the coefficients β ij
using standard model-building techniques frommultiple
regression, and then assess the parameters of the residual
process u ( x ) using, for example, variogram methods on
the estimated residuals from the fitted model. Given
the emulator, we can then carry out the history matching
procedures described above, but, instead of evaluating the
function at each input choice, we evaluate the emulator
expectationE[ f i ( x )] at each chosen x .We therefore need to
add the emulator variance Var[ f i ( x )] to the observational
variance and model error variance terms when making
the comparison between z i and E[ f i ( x )], but otherwise the
analysis is the same as for fast-to-run models.

=

26.3 Assessing model adequacy for a
fast rainfall-runoff model

We consider a rainfall runoff model described in
Iorgulescu et al . (2005), that simulates fluctuations in
water discharge and calcium and sodium concentrations
over time. We illustrate our methods with its application
to a particular sub-catchment of the Haute-Mentue
research catchment (Switzerland) (see Iorgulescu et al .,
2005, who refer to other studies and runoff models).
Each model run simulates three time series: discharge
( D ) and the tracers Calcium (Ca) and Silicon (Si) over