Memory - Computational Explorations in Cognitive Neuroscience - page 307

This prefrontal cortex model has extended our reper-
toire of basic neural mechanisms by relying heavily on
neuromodulation , specifically by dopamine. We have
seen that the network can achieve contradictory objec-
tives (e.g., maintenance and updating) by dynamically
modulating the strengths of different sets of connec-
tions. There are other models in the literature that have
exploited neuromodulators to achieve other contradic-
tory objectives (e.g., storage versus recall in the hip-
pocampus based on the neuromodulator acetylcholine;
Hasselmo & Wyble, 1997). We think that the use of
neuromodulation plays an essential role in transforming
the fairly static and passive network models of informa-
tion processing (e.g., where an input is simply mapped
onto a corresponding output) into more dynamic and
flexible models that more closely approximate the be-
havior of real organisms.
Of course, any time we introduce new mechanisms
such as neuromodulation, we must question whether
they are really essential, or rather constitute a kind of
“hack” or easy solution to a problem that could perhaps
be more elegantly or parsimoniously resolved with a
more careful or judicious application of existing mech-
anisms. We are sensitive to this concern. For example,
in the case of our model of the hippocampus, we did not
find it necessary to include the neuromodulatory mech-
anisms proposed by Hasselmo and Wyble (1997). This
was in part because the natural dynamics of the network
actually end up resolving some of the problems that
Hasselmo and Wyble (1997) had used neuromodulation
for. Nevertheless, we do think that neuromodulation
plays an important role in the hippocampus, especially
as it is used in a more dynamic, controlled-processing
context (see chapter 11 for more discussion).
Another important development reflected in this
model is the use of reinforcement learning for some-
thing that is typically considered to be a high-level cog-
nitive function, working memory. Thus, reinforcement
learning may have a much wider application beyond its
prototypical role in relatively low-level classical con-
ditioning. Interestingly, the more discrete, trial-and-
error nature of reinforcement learning seems to resonate
well with the more explicit, deliberate kinds of learn-
ing strategies that are often associated with higher-level
cognition (e.g., problem solving).

Nevertheless, successful learning in this model
(specifically in the Hidden2 layer) also depended crit-
ically on the kind of incremental, gradient-based learn-
ing that our previous models have emphasized. Thus,
we can see an indication here of how both of these
kinds of learning can cooperate and interact in interest-
ing ways in the acquisition of complex behaviors. Per-
haps one of the most important examples of this learn-
ing has to do with the shaping of the frontal represen-
tations themselves, which is likely a product of both
gradient-based and more discrete reinforcement mech-
anisms. Again, we will follow up on these ideas more
in chapter 11.
Finally, we should note that this model raises as many
questions as it answers (if not more). For example, is it
really possible to explain all updating of active mem-
ory in terms of differences in predicted future reward,
as is the case in this model? Perhaps this basic mech-
anism has been coopted and generalized (especially in
humans?) to allow for dynamic working memory up-
dating in a manner that is not so directly tied to reward
prediction?
In addition to this basic functional-level issue, the
model raises a number of questions about the biologi-
cal mechanisms involved. As we mentioned previously,
the basal ganglia could provide an alternative source
of gating instead of, or in addition to, the VTA. Also,
we discussed two possible solutions to the catch-22
problem for learning when to update the active mem-
ory representations — which of these solutions is actu-
ally employed by the brain (or is it some other as-yet-
unimagined solution)?
Some other questions raised by the model include:
where exactly is the neural system that controls the fir-
ing of the VTA? It must be much more complex than the
simple AC unit in our model — how does that complex-
ity affect the learning and performance of the network?
An important objective of this kind of computational
model is to focus empirical research on questions such
as these.