The Optimal Set of Classifiers - Design and Analysis of Learning Classifier Systems - page 140

way as Waterhouse et al. [227, 226] by performing a Laplace approximation of
the posterior.
The Laplace approximation aims at finding a Gaussian approximation to the
posterior density, by centring the Gaussian on the mode of the density and
deriving its covariance by a second-order Taylor expansion of the posterior [19].
The mode of the posterior is found by solving

∂ ln q V ( V )
∂ V

=0 , (7.46)

which, by using the posterior (7.45) and the definition of g k (7.10), results in

( r nk − g k ( x n )) φ ( x ) − E β ( β k ) v k =0 , k =1 ,...,K. (7.47)

n

Note that, besides the addition of the E β ( β k ) v k term due to the shrinkage prior

on v k , the minimum we seek is equivalent to the one of the prior-less generalised
softmax function, given by (6.11). Therefore, we can find this minimum by ap-
plying the IRLS algorithm (6.5) with error function E ( V )=

−

ln q V ( V ), where

the required gradient vector and the D V × D V blocks H kj of the Hessian matrix

(6.9) are given by

⎛ ⎞

∇ v 1 E ( V )
.

⎝ ⎠

∇ v j E ( V )=
n

∇ V E ( V )= , ( g j ( x n ) − r nj ) φ ( x n )+ E β ( β j ) v j ,

∇ v K E ( V )

(7.48)

and

H kj = H jk =
n

g j ( x n )) φ ( x n ) φ ( x n ) T + I kj E β ( β k ) I .

g k ( x n )( I kj −

(7.49)

I kj is the kj th element of the identity matrix, and the second I in the above
expression is an identity matrix of size D V D V . As the resulting Hessian is

positive definite [173], the posterior density is concave and has a unique maxi-
mum. More details on how to implement the IRLS algorithm are given in the
next chapter.
Let V ∗ with components v k denote the parameters that maximise (7.45).
V ∗ gives the mode of the posterior density, and thus the mean vector of its
Gaussian approximation. As the logarithm of a Gaussian distribution is a qua-
dratic function of the variables, this quadratic form can be recovered by a
second-order Taylor expansion of ln q V ( V ) [19], which results in the precision
matrix

×

Λ V ln q V ( V ∗ )= E ( V ∗ )= H

= −∇∇ ∇∇ | V = V ∗ , (7.50)