Neural Abstraction Pyramid Architecture - Hierarchical Neural Networks for Image Interpretation - page 78

2 1

st
lt

p_sig
p_sat

1.5 .75

1 .5

.5 .25

0 0

-2 -1 0 1 2 -4 -2 0 2 4

(a) (b)

x x

Fig. 4.6. Rectifying transfer functions: (a) without saturation ( f lt : α = 1 ; f st : β = 2 ); (b)
with saturation ( f p sat : α = 2 ; f p sig : β = 1 ).

Furthermore, nonlinear units are needed to make decisions. Because the linear
combination of linear units yields a linear function of the inputs, decisions cannot
be made with linear units alone.
Another important class of transfer functions are the rectifying functions, shown
in Figure 4.6(a):

log(1 + e βx )
β

0 : x ≤ 0
αx : x > 0

f lt ( x ) =

and f st ( x ) =

.

The linear threshold function f lt has derivative zero for negative arguments and a
constant derivative α for positive arguments. A smooth approximation to this func-
tion is f st . Its derivative is nonzero everywhere. Such rectifying functions are used
in models that resemble the nonnegative activity of spiking neurons. These models
assume that the growth of activity is limited by strong recurrent inhibition and that
the functionally important nonlinearity of biological neurons is not the saturation of
the firing rate for high input currents, but the muting of spiking neurons when the
net input is inhibitory.
Another possibility to limit the growth of activity is to use saturation for high
input values. The saturating rectifying functions

f p sat ( x ) = max(0 ,f pn sat ) and f p sig ( x ) = max(0 ,f pn sig )
are shown in Figure 4.6(b). They limit their output values to the interval [0 , 1] .
If not only half-rectification is desired, but the full energy of a signal is required,
a combination of square transfer function f sqr ( x ) = x 2

for the projection units and

√

square root f sqrt ( x ) =

x for the output unit can be used. These transfer functions
are illustrated in Figure 4.7(a). They are applied in some models of complex cells,
where two orthogonal orientation-sensitive projections are combined to a phase-
invariant orientation-sensitive response.
Another special-purpose pair of transfer functions, shown in Figure 4.7(b), is
the logarithmic function f log ( x ) = log( x ) , used in the projection units, followed
by an exponential f exp ( x ) = e x in the output unit. Here, it must be ensured that the
argument of the logarithm is positive. These transfer functions lead to the conversion
of the output unit into a

Q

-unit that multiplies powers of the projection sums: