Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 196

3. If J ( w ( i )) <J ( w ( i − 1)), then retrieve w ( i − 1), divide µ i by r and go to

step 2.
4. Otherwise multiply µ i by r . Iterate until a suitable value of µ i is found.

2.10.7 Kullback-Leibler Divergence Between two Gaussians

The expression of the Kullback-Leiibbler divergence between two Gaussians
with mean and standard deviation ( µ 1 , σ 1 )and( µ 2 , σ 2 ) respectively is derived.
The following relations are useful:

exp ( x

d x =1

+ ∞

1
σ √ 2 π

µ ) 2
2 α 2

−

−∞

x exp ( x

d x = µ

+ ∞

µ ) 2
2 α 2

1
σ √ 2 π

−

−∞

+ ∞

µ )exp ( x

d x = σ 2 .

µ ) 2
2 α 2

1
σ √ 2 π

−

( x −

−∞

The Kullback-Leibler divergence is defined as
D ( p 1 ,p 2 )= + ∞
−∞

p 1 ( x )ln p 1 ( x )
p 2 ( x )

d x.

Because that definition is not symmetrical with respect to the two distri-
butions, the following quantity is preferred:

∆=[ D ( p 1 ,p 2 )+ D ( p 2 ,p 1 )] / 2

+ ∞

exp ( x − µ 1 ) 2
2 σ 1

1
σ 1 √ 2 π

D ( p 1 ,p 2 )=

−∞

ln σ 2 d x

µ 1 ) 2
2 σ 1

µ 2 ) 2
2 σ 2

( x −

+ ( x

−

×

σ 1 −

+ ∞

exp ( x

ln σ 2

µ 1 ) 2
2 σ 1

1
σ 1 √ 2 π

−

=

σ 1 d x

−∞

+ ∞

exp ( x

( x

µ 1 ) 2
2 σ 1

µ 1 ) 2
2 σ 1

− −

− d x

−∞

exp ( x

( x

d x + + ∞

−∞

µ 1 ) 2
2 σ 1

µ 2 ) 2
2 σ 2

− −

The first two terms are equal to ln( σ 2 /σ 1 ) − (1 / 2).

For the third term, one writes

µ 2 ) 2 =( x µ 2 ) 2

( x − − µ 1 + µ 1 −

µ 1 ) 2 +( µ 1 − µ 2 ) 2 +2( x

=( x − − µ 1 )( µ 1 − µ 2 ) .