Information Technology Reference
In-Depth Information
4.2.4 Various Transfer Functions
Both the projection units and the output units of the basic processing element are
equipped with transfer functions
φ
kl
and
ψ
kl
, respectively. These functions deter-
mine the interval of possible projection potentials and cell activities. They are also
the only source of nonlinearity in the Neural Abstraction Pyramid.
The simplest transfer function is the identity:
f
id
(
x
) =
x
. It is used if no non-
linearity and no scaling is desired. Since it does not limit its output values, the iden-
tity transfer function is frequently used for the projection units only. The choice of
φ
kl
P
-unit. In this case, the
weights of the individual projections are treated as if they would contribute directly
to the weighted sum of the output unit.
Common choices for the output transfer function
ψ
kl
are functions that limit the
activities to a finite interval. For instance, the functions
=
f
id
reduces the basic processing element to a single
8
<
:
x
≤−
α
0
1
1 +
e
−βx
1
2
x
2
α
:
−
α < x < α
f
sat
(
x
) =
+
and
f
sig
(
x
) =
:
:
x
≥
α
1
limit the outputs to the interval
[0
,
1]
and the functions
8
<
:
−
1
:
x
≤−
α
2
1 +
e
−βx
x
α
:
−
α < x < α
−
1
f
pn sat
(
x
) =
and
f
pn sig
(
x
) =
:
x
≥
α
1
limit the outputs to the range
[
−
1
,
1]
. The graphs of these functions are drawn in
Figure 4.5. While the piecewise-linear saturation functions
f
sat
and
f
pn sat
have
a derivative of zero outside the interval
[
−
α,α
]
, the sigmoidal functions
f
sig
and
f
pn sig
have a derivative that is nonzero everywhere. This property is important when
an error-signal must be backpropagated.
The use of such nonlinear functions is crucial for the stability of the network
dynamics. When the activity of a unit is driven towards saturation, the effective gain
of the transfer function is reduced considerably. This avoids the explosion of activity
in the network.
1
1
.75
.5
.5
0
.25
sig
sat
-.5
pn_sig
pn_sat
0
-1
-4
-2
0
2
4
-4
-2
0
2
4
(a)
(b)
x
x
Fig. 4.5.
Saturating transfer functions: (a) limiting the activities to the interval
[0, 1]
(
f
sat
:
α = 2
;
f
sig
:
β = 1
); (b) limiting the activities to
[−1, 1]
(
f
pn sat
:
α = 2
;
f
pn sig
:
β = 1
).
Search WWH ::
Custom Search