Unsupervised Learning - Hierarchical Neural Networks for Image Interpretation - page 105

∆w pq
k max l

= η l · a in · a out ,

(5.2)

with a in = H ( I p k max l , J p k max l ) · a i ∗ j ∗ k ∗ l ∗ ,
a out = a ijk max l − a ijk sec l .
A weight is increased by an amount that is proportional to the product of the scaled
input activity a in and the amount a out by which the activity of the winning feature
cell exceeds the one of the second best feature. The use of the difference of the
two most active features instead of the cell activity has a decorrelating effect on the
features. They are forced to respond to different stimuli. How the address i ∗ j ∗ k ∗ l ∗
of the source feature cell is determined is explained in Section 4.2.2.
Because more example windows are available to determine the lower-layer fea-
tures than for the higher-layer ones, the learning rate η l increases with height, e.g.
η l = 0 . 001 K l , where K l is the number of excitatory features in layer l .
The scaling factor H ( I p k max l , J p k max l ) used for the input activity depends on the
offset of a weight relative to the position ( Υ ll ∗ ( i ) ,Υ ll ∗ ( j )) in the source layer l ∗

=
( l − 1) that corresponds to the position ( i,j ) in layer l . H is one in the center of
the window and descends to zero towards the periphery. The weighting enforces a
centered response of the feature cells. This is done because non-centered stimuli can
be represented by neighboring feature cells.
The Hebbian term (5.2) makes the excitatory weights larger. To prevent unlim-
ited weight growth, the sum of the excitatory weights is kept at a value of one by
scaling down all weights by a common factor. The net effect of the normalized Heb-
bian update is that the weights receiving strong input are increased and the other
weights are decreased.

5.2.4 Competition

The normalization of the sum of the weights of excitatory projections, described
above, is a form of competition. The weights compete to receive a large portion of
the projection's constant weight sum.
In addition, competition between the K l excitatory features of layer l is needed
to achieve the desired properties for the produced representation. Care must be taken
that a learning rule enforcing one constraint does not interfere with another rule
enforcing a different constraint.
To fulfill the fairness constraint, the winning frequency of all templates should
be about the same. In order to achieve this, a feature's inhibitory gain I kl is in-
creased each time it wins the competition; otherwise, it is decreased. This makes
features whose winning frequency is above average less active and more specific.
Consequently, these features will not win too frequently in the future. On the other
hand, features that win less often become more active and less specific and there-
fore now win more often. The change is done by adding a small constant ∆

If
kl to
I kl , such that the net effect for a feature that has an average winning frequency is
zero:

: k = k max [winning]

η f If

kl

=

(5.3) ∆

[not winning] , − η f

K l

: k 6 = k max

− 1