Information Technology Reference
In-Depth Information
The model is of the form
g
(
x
,
w
)=
ζ
T
w
=
w
1
+
w
2
x
. The observation
matrix is
⎛
⎞
1
x
1
1
x
2
1
x
3
⎝
⎠
.
Ξ
=
The least squares solution is given by relation
⎛
⎞
⎡
⎛
⎝
⎞
⎤
y
1
p
y
2
p
y
3
p
−
1
w
mc
1
w
mc
2
=
111
x
1
x
2
x
3
111
x
1
x
2
x
3
1
x
1
1
x
2
1
x
3
⎝
⎠
⎣
⎠
⎦
.
.
Clearly, the number of available observations is much too small for a reliable
estimation of the two parameters of the model; this is just a didactic example,
for which geometrical illustrations are feasible.
Geometrical Interpretation
The least squares method has a simple geometrical interpretation, which is
sometimes useful for a better understanding of the results.
We have seen that the vector of the predictions of the model on the training
set can be written as
g
(
ζ
,
w
mc
)=
Ξw
mc
=
Ξ
(
Ξ
T
Ξ
)
−
1
Ξ
T
y
p
.
In observation space (whose dimension is equal to the number of observa-
tions available for training), matrix
Ξ
(
Ξ
T
Ξ
)
−
1
Ξ
T
is the orthogonal projec-
tion matrix onto the subspace spanned by the columns of matrix
Ξ
(called
solution subspace
): thus, the prediction of the model, for a training example,
is the orthogonal projection of the process output onto the solution subspace,
as shown on Fig. 2.5. Note that, among all vectors of solution subspace, the
orthogonal projection of the process output vector is the closest vector to the
process output vector itself: hence, the model obtained by the least squares
solution provides the prediction vector that is closest to the actual output
vector, given the available data.
As an illustration, consider the previous example of a model with one
variable and three observations. The observation space is of dimension 3, and
the subspace spanned by the columns of the observation matrix is of dimension
q
= 2. Figure 2.6 shows the three-dimensional observation space, and the two-
dimensional solution subspace spanned by the vectors
⎛
⎞
⎛
⎞
1
1
1
x
1
x
2
x
3
ζ
1
=
⎝
⎠
and
ζ
2
=
⎝
⎠
.
Search WWH ::
Custom Search