Information Technology Reference
In-Depth Information
where, again for simplicity, we have assumed the same kernel function with the
parameters a
;
h for all the sources and the same number of samples N for each one.
Note that this corresponds to a mixture of Gaussian models where the number of
Gaussians is maximum (one for every observation) and the parameters are equal.
Reducing the pdf of the sources to a standard mixture of Gaussians with a different
number of components and priors for each source does not help in computing the
Kullback-Leibler distance because there is no analytical solution for it. Therefore,
we prefer to maintain the non-parametric approximation of the pdf in order to
model more complex distributions than a mixture of a small finite number of
Gaussians, such as three or four.
The symmetric Kullback-Leibler distance between the clusters u
;
v can be
expressed as:
D
KL
p
x
u
ðÞ==
p
x
v
ðÞ
ð
Þ ¼
H x
ðÞ
H x
ðÞ
Z
p
x
u
ðÞ
log p
x
v
ðÞ
dx
Z
p
x
v
ðÞ
log p
x
u
ðÞ
dx
ð
4
:
7
Þ
where H
ðÞ
is the entropy, which is defined as H
ðÞ¼
E log p
x
ð½ :
To obtain
the distance, we have to calculate the entropy for both clusters as well as the cross-
entropy terms E
x
v
log p
x
u
ðÞ
½ ;
E
x
u
log p
x
v
ð½
The entropy for the cluster u can be calculated through the entropy of the
sources of that cluster taking into account the linear transformation of the random
variables and their independence Eq. (
4.5
):
H x
ðÞ¼
X
M
Hs
u
ðÞþ
log det A
u
j
j
ð
4
:
8
Þ
i
¼
1
The entropy of the sources cannot be analytically calculated. Instead, we can
obtain a sample estimate of
^
Hs
u
ðÞ
using the training data. Denote the i-th source
obtained for the cluster u by
f
s
u
i
ðÞ;
s
u
i
ðÞ;
...
;
s
u
i
Q
ðÞ
g:
The entropy can be
approximated as follows:
h
i
¼
1
Q
i
X
Q
i
^
Hs
u
ðÞ¼
E log p
s
u
i
s
u
ðÞ
log p
s
u
i
ð
s
u
i
ðÞ
Þ
n
¼
1
ð
4
:
9
Þ
2
Þ¼
X
N
su
i
ðÞ
su
i
ðÞ
h
ae
2
p
s
u
i
ð
s
u
i
ðÞ
l
¼
1
The entropy of H x
ðÞ
is obtained analogously:
H x
ðÞ¼
X
j ¼
X
M
M
^
HS
v
ðÞþ
log det A
v
Hs
v
ðÞþ
log det A
v
j
j
j
i
¼
1
i
¼
1
2
1
2
s
v
i
ðÞ
s
v
i
ðÞ
h
X
Q
i
Þ¼
X
N
^
HS
v
ðÞ¼
1
Q
i
log p
s
vi
s
v
i
ðÞ
ð
Þ;
p
s
vi
s
v
i
ðÞ
ð
ae
n
¼
1
l
¼
1
ð
4
:
10
Þ
Search WWH ::
Custom Search