Neural Networks: An Overview - Neural Networks: Methodology and Applications - page 3

most frequently used potential v is a weighted sum of the inputs, with an
additional constant term called “bias”,

v = w 0 + n− 1

w i x i .

i =1

Function f is termed activation function. For reasons that will be explained
below, it is advisable that function f be a sigmoid function (i.e., an s-
shaped function), such as the tanh function or the inverse tangent function.
In most applications that will be described in the present chapter, the out-
put y of a neuron with inputs is given by y = tanh[ w 0 + n− 1

{ x i }

i =1 w i x i ].

• The parameters are assigned to the neuron nonlinearity, i.e., they belong to

the very definition of the activation function such is the case when function
f is a radial basis function (RBF) or a wavelet; the former stem from ap-
proximation theory [Powell 1987], the latter from signal processing [Mallat
1989].
For instance, the output of a Gaussian RBF is given by
y =exp w i ) 2

n

2 w 2
n +1

− ( x i − ,

i =1

where the parameters w i , i =1to n , are the position of the center of the
Gaussian and w n +1 is its standard deviation.
Additional examples of neurons are given in the theoretical and algorithmic
supplements, at the end of the chapter.
For practical purposes, the main difference between the above two cate-
gories of neurons is that RBFs and wavelets are local nonlinearities, which
vanish asymptotically in all directions of input space, whereas neurons that
have a potential and a sigmoid nonlinearity have an infinite-range influence
along the direction defined by v =0.

1.1.1 Neural Networks

It has just been shown that a neuron is a nonlinear, parameterized function of
its input variables. Naturally enough, a network of neurons is the composition
of the nonlinear functions of two or more neurons.
Neural networks come in two classes: feedforward networks and recurrent
(or feedback) networks.

1.1.1.1 Feedforward Neural Networks

General Form

A feedforward neural network is a nonlinear function of its inputs, which is
the composition of the functions of its neurons.