Neurocomputing in Complex Domain - High Dimensional Neurocomputing: Growth, Appraisal and Applications - page 54

Fig. 3.3 a Encircled are the points where Haykin activation and its derivative vanish. Four singular
points of the Haykin Activation Function Eq. 3.1 (explicitly, (0,1), (0, − 1), (0,3), (0, − 3)) have been
fully engulfed by the point scatter. The points (0,3) and (0,

−

3) are shown encircled while the other
two points are completely inside the cloud of points. b Typical convergence pattern with Haykin
activation, shows the peaks formed during a run of training in error-epochs characteristic

the functions are not bounded, they can be used by introducing bounding operation
for their regions. The second type concerns the functions having the bounded singu-
lar points, e.g., the discontinuous functions. These singularities can be removed and
thus they can also be used for activation functions and can achieve their universality.
The last type is for the functions with the so-called essential singularities, i.e., their
singularities cannot be removed. These functions can also be used as activation func-
tions, with the consideration of restricting the regions for them so that their regions
never cover their singularities. But, backpropagation is a week optimization proce-
dure, therefore there exist no mechanism of constraining the values that the weights
can assume. Thus, the value of z that depends upon both the inputs and the weights,
therefore they can take any value on the complex plane. Based on this observation.
the suggested remedy is inadequate and there is a need to find some other activation
functions, which can satisfy the resonable properties of suitable activation function.

3.2.3.3 Two-Dimension Extension of Real Activation Function
(Split-Type Function)

By Liouville's theorem, an analytic function can not be bounded on all of the complex
plane unless it is a constant, therefore the analyticity of the complex function can not
be preserved because boundedness is required, to ensure convergence. Benvenuto and
Piazza [ 17 ] have suggested an alternative complex activation function to avoid the
conflict between the requirements of an activation function and Liouville's theorem,
in general. The characteristic function f C appears in function of real and imaginary
parts of weighted inputs separately (a split-type activation function) in a multilayer
neural network framework: