Memory - Computational Explorations in Cognitive Neuroscience - page 285

a) b)

Hippocampus

A B

HC HC

Input Input

Cortex

A B

Figure 9.11: Conjunctive, pattern-separated representations
result from sparseness. The extreme case where only one re-
ceiving unit (in the upper layer, representing the hippocam-
pus) is allowed to be active is shown here for simplicity. Each
receiving unit has roughly the same number of randomly dis-
tributed connections from the input units. The two shown here
have overlapping input connections, except for one unique
unit each. Thus, two very similar input patterns sharing all
the overlapping units and differing only in these unique units
(shown in panels a and b) will get completely non-overlapping
(separated) memory representations. Thus, the conjunctive
memory representation resulting from sparseness produces
pattern separation.

Figure 9.10: Illustration of pattern separation in the hip-
pocampus. Small gray circles represent units. Circles A and
B in the cortex and hippocampus indicate two sets of repre-
sentations composed of patterns of active units. In the cortex,
they are overlapping, and encompass relatively large propor-
tions of active units. In the hippocampus, the representations
are sparser as indicated by their smaller size, and thus overlap
less (more pattern separation). Units in the hippocampus are
conjunctive and are activated only by specific combinations of
activity in the cortex. atively high (e.g., because the level of inhibition is rel-

atively strong for a given amount of excitatory input).
Figure 9.11 shows how a high inhibitory threshold leads
simultaneously to both pattern separation and conjunc-
tive representations , which are representations that de-
pend critically on the entire conjunction of active units
in the input. The central idea is that sensitivity to the
conjunction of activity in the input produced by a high
threshold leads to pattern separation because even if two
input patterns share a relatively large number of over-
lapping inputs, the overall conjunction (configuration)
of input activity can be different enough to activate dif-
ferent hippocampal units.
A high threshold leads to conjunctive representations
because only those units having the closest alignment of
their weight patterns with the current input activity pat-
tern will receive enough excitation to become activated.
In other words, the activation a unit receives must be a
relatively high proportion of the total number of input
units that are active, meaning that it is the specific com-
bination or conjunction of these inputs that are respon-
sible for driving the units. Figure 9.11 illustrates this
effect in the extreme case where only the most excited
receiving unit gets active. In reality, multiple (roughly

some fixed probability of a unit getting active. In this
case, if you have fewer units active, the odds that the
same units will be active in two different patterns will
go down (figure 9.10). For example, if the probability
of getting active for one pattern (i.e., the sparseness) is
.25, then the probability of getting active for both pat-
terns would be :25 2 or .0625. If the patterns are made
more sparse so that the probability is now .05 for being
active in one pattern, the probability of being active in
both patterns falls to .0025. Thus, the pattern overlap is
reduced by a factor of 25 by reducing the sparseness by
a factor of 5 in this case. However, this analysis assumes
that units are activated at random, ignoring the fact that
they are actually driven by weighted connections with
the input patterns.
A more complete understanding of pattern separation
can be achieved by considering the concept of a unit's
activation threshold — how much excitation it requires
to overcome the inhibitory competition from other units
(Marr, 1969; O'Reilly & McClelland, 1994). To pro-
duce sparse representations, this threshold must be rel-