ICA and ICAMM Methods - On Statistical Pattern Recognition in Independent Component Analysis Mixture Modelling - page 41

" #

L 0 ð W Þ¼ X

X

M M

1
Nh

s i s il
h

E log j

i ¼ 1 l ¼ 1

" ! # ð 2 : 26 Þ

X X X

M M M

w i x ð k Þ x ð l Þ

1
N

1
Nh

log j

h

i ¼ 1 k ¼ 1 l ¼ 1

The overall optimization problem can thus be posed as

" ! # log j det W j

X

M

X

M

X

M

w i x ð k Þ x ð l Þ

1
N

1
Nh

min
w

log j

h

ð 2 : 27 Þ

i ¼ 1 k ¼ 1 l ¼ 1

s : t jj w i jj ¼ 1 ; i ¼ 1 ; ... ; M ð 2 : 28 Þ

Given the sample data x ð k Þ ; k ¼ 1 ; ... ; N ; the objective of Eq. ( 2.27 ) is a smooth
nonlinear function of the elements of the matrix W. The additional constraints of
Eq. ( 2.28 ) restrict the space of possible solutions of the problem to a finite set. The
optimization technique applied is the quasi-Newton method.

2.3.2 Radical

The Radical algorithm [ 50 ] uses entropy minimization, i.e., it must estimate the
entropy of each marginal for each possible W matrix. The Radical marginal
entropy estimates are functions of the order statistics of those marginals.
The order statistics are estimated using spacings estimates of entropy. Consider
a one-dimensional random variable Z, and a random sample of Z denoted by
Z 1 ; Z 2 ; ... ; Z N . The order statistics of a random sample of Z are simply the ele-
ments of the sample rearranged in non-decreasing order: Z ð 1 Þ Z ð 2 Þ Z N .A
spacing of order m,orm spacings is then defined to be Z ð i þ m Þ Z N ; for
1 i i þ m N : Finally, if m is a function of N,am N spacings such as
Z ð i þ m Þ Z ð i Þ ; can be defined.
For any random variable Z with an impulse-free density p ðÞ and continuous
distribution function p x = C k

p s ðÞ , the following holds. Let p be the
Z-way product density p Z 1 ; Z 2 ; ... ; Z N

Þ¼ det A 1
k

ð

Þ ¼ pZ ð pZ ð ...pZ ð .Then

ð

h PZ ð i Þ i ¼

1
N þ 1 ; 8 i ; 1 i N 1

E p PZ ð i þ 1 Þ

ð 2 : 29 Þ

Using these ideas, the following simple entropy estimator can be derived.