Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 87

Hidden2

able to overcome the relatively strong level of leak cur-
rent, whereas the unit that does not have these weights
suffers significantly from it.
Finally, note that this (and the preceding simulations)
are highly simplified, to make the basic phenomena and
underlying mechanisms clear. As we will see when we
start using learning algorithms in chapters 4-6, these
networks will become more complex and can deal with
a large number of intertwined patterns encoded over the
same units. This makes the resulting behavior much
more powerful, but also somewhat more difficult to un-
derstand in detail. Nevertheless, the same basic princi-
ples are at work.

top−down

bottom−up

Hidden1

Input

Figure 3.15: Bidirectionally connected network with
bottom-up and top-down connections between the two hidden
layers.

Go to the PDP++Root window. To continue on to
the next simulation, close this project first by selecting
.projects/Remove/Project_0 . Or, if you wish to
stop now, quit by selecting Object/Quit .

strap comes from the notion of pulling oneself up by
one's own bootstraps — making something from noth-
ing). We will also see how amplification can cause un-
controlled spread of activation throughout the network.

3.4.3 Bidirectional Amplification

Exploration of Simple Top-Down Amplification

The above examples involving bidirectional connectiv-
ity have been carefully controlled to avoid what can be
one of the biggest problems of bidirectional connectiv-
ity: uncontrolled positive feedback . However, this pos-
itive feedback also gives rise to the useful phenomenon
of amplification , where excitatory signals reverberat-
ing between neurons result in dramatically enhanced ac-
tivation strengths. One can think of amplification as just
a more extreme example of the kind of mutual support
or resonance explored in the previous simulation.
Amplification is critical for explaining many aspects
of cognition, including the word-superiority effect dis-
cussed in the introduction (section 1.6.3), and others
that we encounter later. Positive feedback becomes
a problem when it results in the uncontrolled spread
of excitation throughout the entire network, producing
what is effectively an epileptic seizure with virtually
every unit becoming activated. The inhibitory mech-
anisms described in the next section are necessary to
take full advantage of bidirectional amplification with-
out suffering from these consequences.
In this section, we will explore a couple of simple
demonstrations of amplification in action, and see how
it can lead to the ability to bootstrap from weak ini-
tial activation to a fully active pattern (the term boot-

We begin by exploring the simple case of the top-down
amplification of a weak bottom-up input via bidirec-
tional excitatory connections.

Open the project amp_top_down.proj.gz in
chapter_3 to begin.
You will see a network with 3 layers: an input, and 2
bidirectionally connected hidden layers ( hidden1 and
hidden2 ), with 1 unit in each layer (figure 3.15).

You can explore the weights using r.wt as before.
Return to act when done.
Now, we can pull up a graph log to plot the activa-
tions of the two hidden-layer units over time as the net-
work settles in response to the input.

Press View in the amp_td_ctrl control panel and
select GRAPH_LOG . Then, hit Run .
You should see something like figure 3.16 in your
graph window. The activation coming top-down from
the hidden2 unit is amplifying the relatively weak ini-
tial activation of the hidden1 unit, resulting in the
strong activation of both units. This is an excellent ex-
ample of bootstrapping , because the hidden1 unit has
to activate the hidden2 unit in the first place before it
can receive the additional top-down excitation from it.
Let's test the effects of the leak current.