Learning Mixtures of Independent Component Analysers - On Statistical Pattern Recognition in Independent Component Analysis Mixture Modelling - page 60

Table 3.1 Steps for iterative solutions in ICAMM

0. Initialize i ¼ 0 ; W k ð 0 Þ; b k ð 0 Þ

k ¼ 1...Kn ¼ 1...N

1.

Compute s ð n k ð i Þ¼ W k ð i Þ x ð n Þ b k ð i Þ

h i

2.

d log j det W k ð i Þj p s ð n k ð i Þ

Compute d log p x ð n Þ = C k ; W k ð i Þ¼ k ¼ 1...Kn ¼ 1...N.

dW k dW k

p ð C k Þ

3.

j det W k ð i Þj p s ð n k ð i Þ

ð i Þ¼

Compute pC k = x ð n Þ ; W k ¼ 1...Kn ¼ 1...N

P

K

j det W k 0 ð i Þj p s ð n Þ

k 0 ð i Þ

k 0 ¼ 1

4.

Compute dL ð X = W Þ
dW k

ð i Þ k ¼ 1...K using

¼ X

N

d log p x ð n Þ = C k ; W k

dL ð X = W Þ
dW k

pC k = x ð n Þ ; W k ¼ 1...K and the results of steps

dW k

n ¼ 1

2 and 3

5. Actualize W k ; b k k ¼ 1...K using a gradient algorithm

W k ð i þ 1 Þ¼ W k ð i Þþ a dL ð x = W Þ
dW k

b k ð i þ 1 Þ¼ b k ð i Þþ b dL ð x = W Þ
db k

ð i Þ ; ð i Þ k ¼ 1...K

(3.6)
where higher values a and b increase the speed of convergence and the final error variance

6. Go back to step 1, with the new values W k ð i þ 1 Þ; b k ð i þ 1 Þ and i ! i þ 1

2

km s ð n 0 Þ km

s ð n Þ

¼ a X
n 0 6¼ n

2

h

ps ð n Þ
kM

e ; m ¼ 1...Mk ¼ 1...K ð 3 : 7 Þ

where a is a normalization constant and h is a constant that defines the degree of
smoothing of the estimated pdf. Equation ( 3.7 ) must be applied at every iteration
of the algorithm on the source training sets computed in step 2. Using Eq. ( 3.7 ), we
can finally write [ 5 ]:

p ð C k Þ

j det W k j p s ð n Þ

x ð n Þ b k T

¼ X

N

1 þ fs ð n Þ
k

dL ð X = W Þ
dW k

k

W k

P

K

j det W k 0 j p s ð n Þ

n ¼ 1

k 0

k 0 ¼ 1

ð 3 : 8a Þ

p ð C k Þ

j det W k j p s ð n Þ
k

h h i w km i

¼ X

N

dL ð X = W Þ
db k

diag fs ð n Þ
k

ð 3 : 8b Þ

P

K

j det W k 0 j p s ð n Þ

n ¼ 1

k 0

k 0 ¼ 1

Another possibility is replacing the result of Eq. ( 3.8a ) by the natural gradient.
This method of optimization has demonstrated good convergence properties [ 7 ].
Thus we can write