Networks of Neurons - Computational Explorations in Cognitive Neuroscience - page 81

Although we used localist units in our digit network
example, we will see in a moment that distributed rep-
resentations have the same capacity to implement trans-
formations of the input pattern, where these transfor-
mations emphasize some distinctions and deemphasize
others. Using a feature-based notion of distributed rep-
resentations, we can imagine that by detecting some
features and not others, the resulting activity pattern
over the hidden layer will emphasize distinctions based
on these features, and it will deemphasize distinctions
based on features that were not encoded.
The main advantages of distributed representations
over localist ones are as follows (Hinton et al., 1986):

Efficiency: Fewer total units are required to represent
a given number of input stimuli if the representation
is shared across units, because individual stimuli can
be represented as different combinations of unit ac-
tivities, of which there are a very large number for
even modest numbers of units. In contrast, one unit
per pattern is required for localist representations.
Similarity: Distributed representations provide a nat-
ural means of encoding the similarity relationships
among different patterns as a function of the number
of units in common ( pattern overlap ).
Generalization: A network with distributed represen-
tations can often respond appropriately to novel input
patterns by using appropriate (novel) combinations of
hidden units. This type of generalization is impossi-
ble for localist networks, because they require the use
of an entirely new unit to represent a novel input pat-
tern.
Robustness: Having multiple units participating in
each representation makes it more robust against
damage, since there is some redundancy.
Accuracy: In representing continuous dimensions, dis-
tributed ( coarse coded ) representations are much
more accurate than the equivalent number of local-
ist representations, because a lot of information is
contained in the relative activities of the set of active
units. Even in the binary case, a distributed represen-
tation over n units can encode as many as 2 n differ-
ent values, whereas localist units can only represent

Figure 3.12: Tuning functions of a cortical neuron in re-
sponse to visual shapes, showing the distributed nature of
these representations — the neuron responds to a range of
different shapes (and multiple neurons respond to the same
shape, though this is not represented here). The neural re-
sponse is shown below the corresponding visual image that
was presented to a monkey. The neuron was located in cortical
area IT (specifically TE). Reproduced from Tanaka (1996).

represent a subcomponent or feature of the stimulus,
known as a feature-based representation. Of course,
as with the notion of detectors, a distributed representa-
tion can be effective without needing clearly defined or
easily described featural representations. Indeed, in the
case of coarse coded representations of continuous di-
mensions (see section 1.6.2), the “features” are just ar-
bitrary ranges of the underlying dimension over which
the unit gives a graded response.
It is important to emphasize that the issue of whether
one can provide some kind of simple interpretation of
what a given unit seems to encode is in principle un-
related to the distinction between localist versus dis-
tributed representations. Thus, it is possible to have
feature-based distributed representations with easily de-
scribed featural encodings, or localist representations
where it is difficult to describe the representations. Nev-
ertheless, localist representations are most useful (and
typically used) in cases where one can easily label the
units, and often it is difficult to determine exactly what
individual units in a distributed representation encode.

different values (at an accuracy of 1=n ).