Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 74

activity over the input. We think of this transformation
process as altering the similarity structure of the input
activity patterns — making patterns more or less similar
to each other. Specifically, a given transformation will
tend to emphasize some distinctions among patterns,
while deemphasizing or collapsing across others.
Thus, to understand the transformation process, we
need to take into account the activities of multiple de-
tectors , and how these activities vary over multiple dif-
ferent input patterns . We will extend our use of the term
representation from the individual detector case to this
multiple detector case by using it to refer to the proper-
ties of the pattern of activity over a set of multiple de-
tector units, and across a set of different input patterns.
Thus, this notion of representation is much more com-
plex than the simple idea that a representation is what a
detector detects. We will see however that there is some
relationship between these two uses of the term.
To build up to the case of multiple detectors, we can
start with the simpler example of the single '8' detec-
tor explored in section 2.6.3. Here, the patterns of ac-
tivity over the input images (representing the different
digits in a 5x7 grid) were transformed into a pattern of
activity of the detector unit over the different input pat-
terns. This activity pattern reflected the extent to which
each input pattern resembled an 8. Thus, the specific
weight values of this detector imposed a transformation
that emphasized similarity to the 8 pattern, while at the
same time deemphasizing many of the other different
ways in which the input patterns could be considered
similar (e.g., according to whether they have a particu-
lar feature element like a vertical or horizontal line in a
particular location).
The example that we explore here extends this single
8-detector network to the case where there are multiple
units in the second layer (i.e., a hidden layer) contain-
ing detectors for all of the different digits (figure 3.6).
Thus, when an image of any digit is presented, the unit
in this digit network that represents that digit would be
activated.
To see how this network transforms the input, we
have to consider in general all possible input patterns
that could be presented to the network. Specifically,
there will be inputs that look like noisy versions of each
of the digits (figure 3.7). If a noisy version is sufficiently

5 6 7 8 9

0 1 2 3 4

Hidden

Input

Figure 3.6: The digit network, with an input layer for digit
images, and a hidden layer with units that represent the differ-
ent digits.

Figure 3.7: Noisy versions of an “8” input image (the origi-
nal and weight template is the first image).

similar to the weight pattern of a digit detector, then that
detector will fire. Importantly, just by looking at the ac-
tivity of the hidden layer units, we will not in general
be able to tell the difference between any of many dif-
ferent input patterns that fire a given detector. If we in-
stead looked directly at the input pattern, we could tell
the differences between the different noisy versions.
Thus, the hidden layer detectors have deemphasized
the distinctions among all the noisy versions of the dig-
its, while at the same time emphasizing the distinctions
between the different digits (which are represented by
different units). Although many other possible kinds of
transformations could be implemented by configuring
the detector weights in different ways, this particular
type of transformation is beneficial if you happen to be
concerned with the identity of the digit in question (e.g.,
for the purposes of doing mathematical calculations),
and not the image details of the particular rendition of
that digit (e.g., the “noisy” versions could amount to
differences in font, color, size, etc.). We can think of