Information Technology Reference
In-Depth Information
Learning:
Distributed representations allow for the
bootstrapping of small changes that is critical for
learning (see section 1.6.5 and chapters 4-6). Fur-
thermore, learning tends not to result in perfectly
interpretable representations at the single unit level
(i.e., not template matching units).
these units will be active whenever one of these features
is present in the input.
Press
Run
.Do
View
,
EVENTS
to see the input pat-
terns.
Now verify for yourself in the GridLog that the firing
patterns of the hidden units make sense given the fea-
tures present in the different digits. The only case that
is somewhat strange is the third hidden unit firing for
the digit “0” — it fires because the left and right sides
match the weight pattern. There is an important lesson
here — just because you might have visually encoded
the third hidden unit as the “middle horizontal line” de-
tector, it can actually serve
multiple roles
. This is just
a simple case of the kind of complexity that surrounds
the attempt to describe the
content
of what is being de-
tected by neurons. Imagine if there were 5,000 weights,
with a much more complicated pattern of values, and
you can start to get a feel for how complicated a neu-
ron's responses can be.
Now, let's see what a cluster plot of the hidden unit
representations tells us about the properties of the trans-
formation performed by this distributed network.
,
!
As should be clear, distributed representations are
critical for many of the key properties of neural net-
works described in the introductory chapter. Despite
this, some researchers maintain that localist representa-
tions are preferable, in part for their simplicity, which
was admittedly useful in the previous section. Never-
theless, most of the remaining models in this topic em-
ploy distributed representations. In section 3.5 we also
discuss the important difference between
sparse dis-
tributed
representations and generic distributed repre-
sentations.
3.3.3
Exploration of Distributed Representations
Now, we will explore the difference between localist
and distributed representations.
Select
NOISY_DIGITS
for
the
env_type
Open the project
loc_dist.proj.gz
in
chapter_3
to begin.
This project looks very similar to the previous one —
it is in fact a superset of it, having both the previous lo-
calist network, and a new distributed one. To begin, we
will first replicate the results we obtained before using
the localist network.
(
Apply
), and
Run
.
Then, do
Cluster
and choose
,
!
CLUSTER_HIDDEN
.
This should produce a cluster plot like that shown in
figure 3.13. Although there are a couple of obvious dif-
ferences between this plot and the one for the localist
network shown in figure 3.8b, it should be clear that
the distributed network is also generally emphasizing
the distinctions between different digits while deempha-
sizing (collapsing) distinctions among noisy versions of
the same digit.
One difference in the distributed network is that it
sometimes collapsed noisy versions of
different
digits
together (a 2 with the 5's, and a 0 with the 4's), even
though in most cases the different versions of the same
digit were properly collapsed together. It also did not
always collapse all of the different images of a digit to-
gether, sometimes only getting 2 out of 3. The reason
for these problems is that the noisy versions actually
shared more features with a different digit representa-
tion. In most cases where we use distributed represen-
tations, we use a learning mechanism to discover the
individual feature detectors, and learning will usually
Press
View
on the overall control panel (now called
loc_dist_ctrl
) and select
GRID_LOG
. Then do
Run
.
You should see the responses of the same localist net-
work just as before.
,
!
Set
network
to
DISTRIBUTED_NETWORK
instead
of
LOCALIST_NETWORK
, and hit
Apply
.
You will now see the distributed network take the
place of the localist one. This network contains only 5
hidden units. Let's explore this network by examining
the weights into these hidden units.
,
!
Select
r.wt
in the network window, and click on
each of the units.
You will notice that these units are configured to de-
tect parts or
features
of digit images, not entire digits
as in the localist network. Thus, you can imagine that
Search WWH ::
Custom Search