Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 88

1−

principles apply to lateral connectivity. Indeed, one
could simply move the hidden2 unit down into the
same layer as hidden1 , where they would be just like
two parts of one interconnected pattern, and if one were
activated, it would be bootstrapped and amplified by the
other.

0.8−

Hidden1 Act

0.6−

Go to the PDP++Root window. To continue on to
the next simulation, close this project first by selecting
.projects/Remove/Project_0 . Or, if you wish to
stop now, quit by selecting Object/Quit .

0.4−

Hidden2 Act , !

0.2−

0−

Exploration of Amplification with Distributed
Representations

0 20 40 60 80 100 120 140 160 180 200

Figure 3.16: Bootstrapping phenomenon where the activa-
tion of the hidden1 unit is just strong enough to start activat-
ing hidden2 , which then comes back and reinforces hidden1
producing strong activation in both.

The previous example illustrated the benefits of bidirec-
tional excitatory amplification, and how bootstrapping
can occur. However, when distributed representations
are used, such amplification can lead to the activation of
inappropriate units. In particular, the overlapping con-
nections required to implement distributed representa-
tions can allow excitation to spread overzealously. We
will see below how this spread can be checked by in-
hibition, but without that, we must resort to increasing
the leak current to prevent activation spread. The prob-
lem here is, as you saw with the above example, if there
is too much leak current, then it is impossible to boot-
strap the representations into activation in the first place,
so that the benefits of bidirectional excitatory amplifica-
tion are not available. Thus, this example provides some
strong motivation for the next section on inhibitory in-
teractions.

Increase the strength of the leak current g_bar_l
from 3.4 to 3.5, and press Run . Next decrease the
strength of the leak current and observe the effects.
You should see that with a higher leak current (3.5),
the resulting hidden1 activation is now insufficient
to activate hidden2 , and no bootstrapping or am-
plification occurs. With decreases to the leak cur-
rent, the bottom-up activation of hidden1 is relatively
strong, so that the bootstrapping and amplification from
hidden2 are less noticeable.
This simple case of bootstrapping and amplification
provides some insight into the word-superiority effect
studied by McClelland and Rumelhart (1981) and sum-
marized in the introductory chapter. Recall that the ba-
sic effect is that people can recognize letters in the con-
text of words better than letters in the context of non-
words. The puzzle is how word-level information can
affect letter-level processing, when the letters presum-
ably must be recognized before words can be activated
in the first place. Based on this exploration, we can see
how initially weak activation of letters (i.e., the hidden
1 unit) can go up to the word-level representations (i.e.,
the hidden 2 unit), and come back down to bootstrap
and amplify corresponding letter-level representations.
Note that although this example is organized accord-
ing to bottom-up and top-down processing, the same

, !

Open the project amp_top_down_dist.proj.gz
in chapter_3 to begin.
The network here is like that in the previous example,
except that now there are multiple units per layer.

, !

Click on the r.wt button, and then click on the 3
hidden1 units.
Notice that they each receive one corresponding in-
put from the input units (this is called one-to-one con-
nectivity). Notice also that the left and right hidden1
units receive uniquely from the left and right hidden2
units, while the center hidden1 unit receives from
both hidden2 units.

, !

Now click on the the left and right hidden2 units.