Information Technology Reference
In-Depth Information
3.2.2 Properties of Suitable Complex-Valued Activation Function
The performance or approximation offered by neural networks depends to a great
extent on its activation function. Therefore, the study of activation functions and their
derivative is important for its proper choice. In case of RVNN, the chosen activation
function should be a smooth (continuously differentiable) and bounded function;
such as sigmoidal function. When a real domain is extended to the complex domain,
there exist certain difficulties involved in the appropriate choice of an activation
function due to the set of properties that a suitable complex activation function must
possess. Let a complex-valued function be
f
(
z
)
=
f
(
z
,
z
)
+
jf
(
z
,
z
)
where
z
=
z
+
z
A suitable activation function is supposed to possess the following reasonable
properties:
•
f
(
z
)
must be nonlinear in
(
z
and
z
)
.If
f
(
z
)
is linear:
- There is no advantage in using learning algorithm.
- The capabilities of the network are severely limited if
f
(
z
)
is linear.
•
f
(
z
)
is bounded.
- This is true if and only if both
f
(
are bounded.
- Both of these are used during training and even if one of them is unbounded, it
can result in divergence.
z
,
z
)
and
f
(
z
,
z
)
•
f
(
z
)
is such that error
e
=
0 and inputs
z
=
0 implies
∇
W
(
E
)
=
0
- Noncompliance with this condition is undesirable since it would mean that even
in the presence of a
nonzero input
and
nonzero error
, it is still possible that no
learning takes place (
0).
- Both of these are used during training and even if one of them is unbounded, it
can result in divergence.
∇
W
(
E
)
=
The partial derivatives
f
(
,
f
(
,
f
(
and
f
(
•
z
)
z
)
z
)
z
)
exist and are bounded.
- These partial derivatives are used during training and the weights are updated
by amounts proportional to these partial derivatives, therefore they must be
bounded.
•
f
(
z
)
is not entire.
- According to Liouville's Theorem if
f
(
z
)
is entire and bounded on the complex
palne, then
f
(
z
)
is a constant function.
Search WWH ::
Custom Search