Machine Recognition in Complex Domain - High Dimensional Neurocomputing: Growth, Appraisal and Applications - page 141

equation Eq. 7.7 , hence for optimization the entropy of the Y can be simplified as:

M

H ( Y ) = ln p s ( U i ) + ln | W | (7.8)

i = 1

By the definition, the pdf p s of a variable is the derivative of that variable's cdf
f , where f is the pdf of every source signal. Thus,

p s ( U i ) = df ( U i )/ dU i

this derivative is denoted by f ( U i ) = p s ( U i ) ; thus above equation can be rewritten

as:

M

ln f (

h ( Y ) = U i ) + ln | W | (7.9)

i = 1

After obtaining the proper description of entropy in terms of the recovered signals
and the unmixing matrix W, one need a method for finding that Wwhich maximizes
entropy of Y, and which therefore maximize the independence of U. The infomax
algorithm for ICA given by Bell and Sejnowski (1995) states that “if the model pdf
p s matches the pdf p u of the extracted signals then maximizing the joint entropy of
Y also maximize the amount of mutual information between X and Y”.
The estimation of optimal W can be achieved by performing gradient ascent on
the entropy of the output Y with respect to weight matrix W. The gradient ascent can
effectively be performed on h(Y) by iteratively adjusting W as in neural network (in
order to maximize the function h(Y)). One can start by finding the partial derivative
of h(Y) with respect to one scalar element W ij of W. The weight W ij determines the
proportion 1 of the j th mixture in the i th extracted signal U i . Given that U

=

WX
and that every source signal has the same pdf f , the partial derivative of h(Y) with
respect to the ij th element in W is

M

ln f ( ∂ h ( Y ) ∂ U i )

+ ∂

ln | W |

W ij =

(7.10)

∂ ∂ W ij ∂ W ij

i = 1

The interested readers may easily evaluate each of the two derivatives on the right
hand side of Eq. 7.10 in turn. Further simplification and rearrangements of terms
yields:

M

W T − 1

f (

∂ h ( Y ) U i )

W ij = ij +

x j (7.11)

f ( ∂ U i )

i = 1

1 Precisely talking, W ij ascertains the proportion only if the weights that contribute to U i sum to
unity, this is of no significance for our objectives.