Information Technology Reference
In-Depth Information
the above expressions, assume that the centers (means) of the classes lie in the
horizontal axis and are symmetric with respect to the origin (every possible
class configuration can be reduced to this case through shifts and rotations).
The optimal decision boundary is the vertical line
x
1
=0with the optimal
set of parameters [
w
1
w
2
w
0
]=[
c
00]with
c
+
to give the correct class
orientation. Let us analyze SEE solutions of the form
w
=[
w
1
∈
R
00]with
w
1
=0.Wefirstnotethat
H
S
(
w
) is piecewise constant
H
S
(
w
)=
c
1
,w
1
>
0
c
2
,w
1
<
0
(4.63)
where
c
1
→
ln(0
.
5) (swapped labels)
for an increasing distance between the classes. Also,
0 (100 % correct classification) or
c
1
→
H
S
(
w
)=0,thatis,
the vectors
w
are critical points of SEE. Considering two particular class
configurations the Hessian
∇
2
H
S
(
w
) will give us insight about the nature of
∇
w
for
w
1
>
0.
±
50
T
1. Distant classes:
μ
±
1
=[
∇
2
H
S
at
w
is given by
The Hessian matrix
⎡
⎣
⎤
⎦
00
0
0
.
4809
w
1
0
0
00
0
.
3527
w
1
2
H
S
(
w
)=
∇
,
which is a positive semi-definite matrix. The nature of
w
cannot be eval-
uated directly due to the singularity of the Hessian. Using the Taylor
expansion of
H
S
to analyze its behavior in a neighborhood of
w
one gets
≈
h
T
2
H
S
(
w
)
h
.
H
S
(
w
+
h
)
−
H
S
(
w
)
∇
(4.64)
where
h
=[
h
1
h
2
h
3
]
T
is a small increment on
w
.
It is easily seen that there are increments
h
=
h
1
00]such that
2
H
S
(
w
)
h
=0.But
w
+
h
belongs to the positive
w
1
axis where
H
S
is constant. Along any other
h
directions, the quadratic form is posi-
tive. This means that
w
, or more precisely, the whole positive
w
1
axis, is
in fact an entropy minimum.
2. Close classes:
h
T
∇
0
.
50
T
μ
±
1
=[
±
The Hessian now becomes
⎡
⎣
⎤
⎦
00
0
0
.
2641
w
1
0
0
00
−
0
.
1377
w
1
2
H
S
(
w
)=
∇
.
(4.65)
This matrix is indefinite (it has positive and negative eigenvalues). This
means that there are directions such that
w
is a minimum and directions
