A Question of Adequacy of Observations in Variational Data Assimilation - Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications - page 109

background field for the 500 mb geopotential field was required to produce a mean-
ingful upper-air analysis over the ocean. The observations alone were insufficient to
yield a product useful for NWP.
The Bergth orsson-Do os analysis was non-optimal, yet it provided guidance for
one of the first optimal data assimilation methods in meteorology—the optimal
or statistical interpolation method [generally referred to as OI method (See Lewis
et al. 2006 )]. The OI method optimally fit the analysis to both background (forecast
from an earlier time) and observations in accord with relative accuracy of these
inputs. It fundamentally depends on the statistical structure of errors associated
with the background forecast. Problems can develop with OI when the background
error covariance matrix is “almost singular” which occurs when the observations
are clustered together and lack any sense of distributional uniformity—related to
the inadequacy of observations. This situation is made all the worse when the
observational errors are small [See the carefully crafted quotation by J. Purser on
the subject in Lewis and Lakshmivarahan ( 2008 )].
When the four-dimensional data assimilation method using adjoint equations
(4D-Var with Adjoint) arrived on the operational scene in the late 1980s [ LeDimet
and Talagrand ( 1986 ), Lewis and Derber ( 1985 ), and Thacker and Long ( 1988 )], the
minimization of the cost function through “steepest descent” was the philosophy
to find the optimal control vector of the model (initial conditions, boundary
conditions, and physical/empirical parameters)—the control that minimized the
squared departure between forecast and observations under the “strong” dynamical
constraint (exact satisfaction of the dynamical law). The method has esthetic appeal
with its foundation in calculus of variations and it also possesses a utilitarian
component through its efficiency in calculating the gradient of the cost function.
However, in the presence of the complex dynamics of atmospheric flow, it is not
unusual to encounter “flatness” in the geometric structure of the cost function in the
space of control and this poses problems for steepest descent-type algorithms. As
explored by Thacker ( 1989 ), the insufficiency issue benefits from an examination
of the Hessian matrix, the matrix of second derivatives of the cost function with
respect to control. The eigenvalues and associated eigenvectors of the Hessian
about the terminal iterated state reveal the structure of the cost function. The
eigenvectors point along the principal axes of the ellipse of constant cost and the
eigenvalues determine the lengths of the semiaxes. As the eigenvalue approaches
zero, the semiaxis approaches infinity and the Hessian approaches singularity. The
cost function surface should curve upward steeply in directions associated with well-
determined elements of control. When the surface is flat along some directions, this
is a sign of ill-conditioning of the optimization problem—another way of saying the
observations are inadequate for finding the optimal state. For the high-dimension
nonlinear dynamics of weather prediction, it is most challenging to determine the
characteristics of the Hessian in spite of efforts to simplify the problem and make it
tractable ( Lewis et al. 2006 ).
In those cases of ill-conditioning, and where acquisition of more observations is
difficult or impossible, the use of prior knowledge such as climatology or a forecast
from an earlier time is the most reasonable avenue of pursuit to rid the problem of