Language - Computational Explorations in Cognitive Neuroscience - page 355

Event attenti binding dyslexi

network to transcend rote memorization of the text it-
self, and produce representations that will be effective
for processing novel text items.

attention 1 0.415 0.090

binding 0.415 1 0.118

dyslexia 0.090 0.118 1

View the GetWordRF representations for several
other units to get a sense of the general patterns across
units.
Although there clearly is sensible semantic structure
at a local level within the individual-unit representa-
tions, it should also be clear that there is no single, co-
herent theme relating all of the words represented by
a given unit. Thus, individual units participate in rep-
resenting many different clusters of semantic structure,
and it is only in the aggregate patterns of activity across
many units that more coherent representations emerge.
This network thus provides an excellent example of dis-
tributed representation.

, !

Table 10.12: Cosine matrix of the hidden representations for
the words “attention,”“binding,” and “dyslexia”.

By turning selwts off, we ensure that the previ-
ously selected units from “attention” will remain so.
Now, you should be able to see that in several cases, the
strong weights for “binding” overlap with those of the
selected units from “attention.” Thus, these overlapping
units represent both of these concepts, as one might ex-
pect given that they have a clear semantic relationship.
Now let's see what a more unrelated word looks like.

Do SelWord again, and enter “dyslexia” as the
word, and keep the other buttons as they are.
You should now observe considerably less overlap
among the strong weights from “dyslexia” and the se-
lected units from “attention,” which is appropriate given
that these words are not as closely related.
The SelWord process can be performed automati-
cally for two different words by using the PairProbe
button. The first word's weights will be selected, and
the second word's weights will be viewed as color val-
ues.

Distributed Representations via Sending Weights

A more appropriate technique for exploring the nature
of the distributed semantic representations is to look at
the sending weights from a given word to the hidden
layer. This will show the pattern of hidden units that
represent that word, and by doing this for multiple dif-
ferent words, we can get a sense of how many hidden
units the words have in common. In other words, we
can look at distributed pattern overlap as a measure of
the similarity structure (as discussed in chapter 3).

Do some PairProbe s for other word pairs that you
expect to be semantically related in this textbook, and
pairs that you expect to be unrelated.

Press SelWord on the sem_ctrl control panel.
Enter “attention” as the word, and click both the view
and selwts buttons on.
This will allow us to view the weights for the input
unit corresponding to the word “attention,” and it will
select all of the hidden units which have a weight value
greater than the rf_thresh of .5.

, !

Summarizing Similarity with Cosines

Instead of just eyeballing the pattern overlap, we can
compute a numerical measure of similarity using nor-
malized inner products or cosines between pairs of
sending weight patterns (see equation 10.1 from the
dyslexia model in section 10.3). The WordMatrix
button does this computation automatically for a list of
words (separated by spaces), producing a matrix of all
pairwise cosines, and a cluster plot of this distance ma-
trix.

Youhavetoselect s.wt in the network to actually
view the sending weights.
You should see a pattern of hidden units that have
strong weights from this word unit, with the strongest
units selected (indicated by the white dashed line sur-
rounding the unit). By itself, this pattern is not very
meaningful. However, we can now compare the pattern
with that of another, related word.

, !

Do WordMatrix , and enter “attention binding
dyslexia” as the words.

Do SelWord again, but enter in “binding” as the
word this time, and click the selwts button off.

, !

, !