Computation in Neural Nets - Understanding Understanding: Essays on Cybernetics and Cognition - page 93

input perturbations (Dynamic Stability); second, in terms of performance,
i.e., the system's integrity of computation despite perturbations of structure
or function of its constituents (Logical Stability); third, to reach stabilities
in the two former senses despite permanent changes in the system's envi-
ronment (Adaptation). We shall briefly touch upon these points.
5.1. Dynamic Stability
Beurle (1962) in England and Farley and Clark (1962) at MIT were proba-
bly the first to consider seriously the behavior of nets of randomly connected
elements with transfer functions comparable to eq. (47). Both investigated
the behavior of about a thousand elements in a planar topology and a neigh-
borhood connection scheme. Beurle used his network to study computation
with distributed memory. To this end, elements were constructed in such a
way that each activation caused a slight threshold reduction at the site of
activity and made the element more prone to fire for subsequent stimuli. The
system as a whole shows remarkable tendencies to stabilize itself, and it
develops dynamic “engrams” in the form of pulsating patterns. Farley and
Clark's work is carried out by network simulation on the Lincoln Labora-
tory's TX-s computer; and the dynamic behavior resulting from defined
stimuli applied to selected elements is recorded with a motion picture
camera. Since elements light up when activated, and the calculation of the
next state in the network takes TX-2 about 0.5 seconds, the film can be pre-
sented at normal speed and one can get a “feeling” for the remarkable variety
of patterns that are caused by variations of the parameters in the network.
However, these “feelings” are at the moment our best clues to determine our
next steps in the approach to these complicated structures.
Networks composed of approximately one thousand Ashby elements
(see fig. 13) were studied by Fitzhugh (1963) who made the significant
observation that slowly adding connections to the element defines with
reproducible accuracy, a “connectedness” by which the system swings from
almost zero activity to full operation, with a relatively small region of inter-
mediate activity. This is an important corollary to an observation made by
Ashby et al. (1962), who showed that networks composed of randomly con-
nected McCulloch elements with facilitatory inputs only, but controlled by
a fixed threshold, show no stability for intermediate activity, only fit or
coma, unless threshold is regulated by the activity of elements.
In all these examples, the transfer function of the elements is varied
in some way or another in order to stabilize the behavior of the system.
This, however, implies that in order to maintain dynamic stability one
has to sacrifice logical stability, for as we have seen in numerous examples
(e.g., fig. 10) variation in threshold changes the function computed by
the element. Hence, to achieve both dynamic and logical stability it is nec-
essary to consider logically stable networks that are immune to threshold
variation.