Background - Design and Analysis of Learning Classifier Systems - page 26

How classifiers are combined to form the global prediction is essential to func-
tion approximation but has been mostly ignored since the initial introduction
of XCS. Only recently, new light has been shed on this component [83, 29], but
there is certainly still room for advancing its understanding.

2.4.4 Approaches from the Reinforcement Learning Side

Again concentrating on XCS, its exact approach to performing reinforcement
learning has been discussed by Lanzi [138] and Butz, Goldberg and Lanzi [45].
In the latter study, Butz et al. show the parallels between XCS and Q-Learning
and aim at adding gradient descent to XCS's update equations. This modification
is additionally published in [47], and was later analysed many times [223, 224,
142, 79, 140, 139], but with mixed results. Due to the current controversy about
this topic, its detailed discussion to Sect. 9.3.6.
Another study that is directly relevant to RL is the limits of XCS in learning
long sequences of actions [11, 12]. As this limitation emerges from the type of
classifier set model that XCS aims at, it is also relevant to this work, and thus
will be discussed in more detail in Sect. 9.5.1.
There has been no work on the stability of XCS when used for sequential
decision tasks, even though such stability is not guaranteed (for example, [25]).
Wada et al. claim in [223, 224] that XCS does not perform Q-Learning correctly -
a claim that is question in Sect. 9.3.6 - and consequently introduce a modification
of ZCS in [224] that makes it equivalent to Q-Learning with linear function
approximation. They demonstrate its instability in [222], and present a stable
variant in [224]. As described in Sect. 4.6, their LCS model is not compatible
with XCS, as they do not train their classifiers independently. For an XCS-like
model structure, stability considerations are discussed in Sect. 9.4.

2.5 Discussion and Conclusion

LCS have come a long way since their initial introduction, and still continue to
be improved. From this historical overview of LCS and in particular XCS we
can see that LCS are traditionally approached algorithmically and also analysed
as such. Even in the first LCS, CS-1, most of the emphasis is put on how to
approach the problem, and little on the problem itself. Given that many non-
LCS approaches handle the same problem class (for example, [17, 209]), an
algorithmic description of LCS emphasises the features that distinguishes LCS
from non-LCS methods. But even with such statements one needs to be careful:
considering the series of 11 short essays under the title “What is a Learning
Classifier System?” [115] it becomes clear that there is no common agreement
about what defines an LCS.
Based to these essays, Kovacs discusses in [134] if LCS should be seen as GA's
or algorithms that perform RL. He concludes that while strength-based LCS are
more similar to GA's, accuracy-based LCS shift their focus more towards RL.
Thus, there is no universal concept that applies to all LCS,particularlywhen