The Optimal Set of Classifiers - Design and Analysis of Learning Classifier Systems - page 155

The data likelihood is, similarly to (7.6), given by

X , W , Z )=
n

w k ) z nk .

p ( Y | p ( y n | (7.118)

k

The mixing model is equivalent to that of the Bayesian LCS model for regression
(see Table 7.1).

7.5.2 Variational Posteriors and Moments

The posteriors are again evaluated by variational Bayesian inference. Starting
with the individual classifiers, their variational posterior is found by applying
(7.24) to (7.112), (7.113), (7.117) and (7.118), and for classifier k results in

q w ( w k )=Dir( w k | α k ) ,

(7.119)

with

α k = α +
n

r nk y n . (7.120)

α k )= n r nk y n / n r nk results in the same fre-
quentist probability estimate as the maximum likelihood procedure described
in Sect. 5.5.2. The prior α acts like additional observations of particular classes.
The variational posterior of Z is the other posterior that is influenced by the
classifier model. Solving (7.24) by combining (7.12), (7.112), (7.117) and (7.118)
gives

Assuming α = 0 , E ( w k |

q Z ( Z )=
n

ρ nk
j ρ nj

r z nk
nk

, with r nk = = E Z ( z nk ) , (7.121)

k

where ρ nk satisfies

| v k = v k +
j

ln ρ nk =ln g k ( x n ) y nj E W (ln w kj )

| v k = v k +
j

( α kj ) ( α k ) .

=ln g k ( x n ) y nj ψ − ψ (7.122)

α k , is, as before, the sum of the elements of α k .
The variational posteriors of V and β remain unchanged, and are thus given
by (7.51) and (7.55).

7.5.3 Variational Bound

For the classification model, the variational bound L ( q )isgivenby

L ( q )= E W,Z (ln p ( Y | X , W , Z )) + E W (ln p ( W )) (7.123)

+ E V (ln p ( Z | X , V )) + E V,β (ln p ( V | β )) + E β (ln p ( β ))

− E W (ln q ( W )) − E V (ln q ( V )) − E β (ln q ( β )) − E Z (ln q ( Z )) .