Image Processing Reference
In-Depth Information
9.2
Appendix 2: Least squares analysis
9.2.1
The least squares criterion
The
least squares criterion
is one of the foundations of
estimation theory
. This is the theory
that concerns extracting the
true
value of signals from
noisy
measurements. Estimation
theory techniques have been used to guide Exocet missiles and astronauts on moon missions
(where navigation data was derived using sextants!), all based on techniques which employ
the least squares criterion. The least squares criterion was originally developed by Gauss
when he was confronted by the problem of measuring the six parameters of the orbits of
planets, given astronomical measurements. These measurements were naturally subject to
error, and Gauss realised that they could be combined together in some way in order to
reduce a best estimate of the six parameters of interest.
Gauss assumed that the noise corrupting the measurements would have a
normal
distribution
, indeed such distributions are often now called Gaussian to honour his great
insight. As a consequence of the
central limit theorem
, it may be assumed that many real
random noise sources are normally distributed. In cases where this assumption is not valid,
the mathematical advantages that accrue from its use generally offset any resulting loss of
accuracy. Also, the assumption of normality is particularly invaluable in view of the fact
that the output of a system excited by Gaussian-distributed noise is Gaussian-distributed
also (as seen in Fourier analysis, Chapter 2). A Gaussian probability distribution of a
variable
x
is defined by
)
2
-(
xx
-
1
2
2
(9.13)
px
() =
e
2
is the second moment
or variance of the distribution. Given many measurements of a single unknown quantity,
when that quantity is subject to errors of a zero-mean (symmetric) normal distribution, it
is well known that the best estimate of the unknown quantity is the average of the
measurements. In the case of two or more unknown quantities, the requirement is to
combine the measurements in such a way that the error in the estimates of the unknown
quantities is minimised. Clearly, direct averaging will not suffice when measurements are
a function of two or more unknown quantities.
Consider the case where
N
equally precise measurements,
f
1
,
f
2
. . .
f
N
, are made on a
linear function
f
(
a
) of a single parameter
a
. The measurements are subject to zero-mean
additive Gaussian noise
v
i
(
t
) as such the measurements are given by
f
i
=
f
(
a
) +
v
i
(
t
)
where
x
is the mean (loosely the average) of the distribution and
i
1,
N
(9.14)
f
between the true value of the function and the noisy measurements of
The differences
it are then
f
= (
f
af
) -
∀ ∈
i
1,
N
(9.15)
i
i
By Equation 9.13, the probability distribution of these errors is
˜
)
2
-(
f
i
1
2
˜
2
pf
() =
e
∀ ∈
i
1,
N
(9.16)
i
