Neural Networks: An Overview - Neural Networks: Methodology and Applications - page 5

General Form

That network computes N o functions of the input variables of the network;
each output is a nonlinear function (computed by the corresponding output
neuron) of the nonlinear functions computed by the hidden neurons.
A feedforward network with n inputs, N c hidden neurons and N o output
neurons computes N o nonlinear functions of its n input variables as composi-
tions of the N c functions computed by the hidden neurons.
It should be noted that feedforward networks are static; if the inputs are
constant, then so are the outputs. The time necessary for the computation
of the function of each neuron is usually negligibly small. Thus, feedforward
neural networks are often termed static networks in contrast with recurrent
or dynamic networks, which will be described in a specific section below.
Feedforward multilayer networks with sigmoid nonlinearities are often
termed multilayer perceptrons, or MLPs .
In the literature, an input layer and input neurons are frequently men-
tioned as part of the structure of a multilayer perceptron. That is confusing
because the inputs (shown as squares on Fig. 1.2, as opposed to neurons,
which are shown as circles) are definitely not neurons: they do not perform
any processing on the inputs, which they just pass as variables of the hidden
neurons.

Feedforward Neural Networks with a Single Hidden Layer of Sigmoids and a
Single Linear Output Neuron

The final part of this presentation of feedforward neural networks will be de-
voted to a class of feedforward neural networks that is particularly important
in practice: networks with a single layer of hidden neurons with a sigmoid
activation function, and a linear output neuron (Fig. 1.3).
The output of that network is given by

⎡ ⎛ ⎞ ⎤

g ( x, w )= N c

n

⎣ w N c +1 ,i tanh ⎝ ⎠ ⎦ + w N c +1 , 0

w ij x j + w i 0

i =1 j =1

⎡ ⎛ ⎞ ⎤

N c n

⎣ w N c +1 ,i tanh ⎝ ⎠ ⎦ + w N c +1 , 0 ,

= w ij x j

i =1 j =0

where x is the input ( n +1)-vector, and w is the vector of ( n +1) N c +( N c +1)
parameters. Hidden neurons are numbered from 1 to N c , and the output
neuron is numbered N c + 1. Conventionally, the parameter w ij is assigned
to the connection that conveys information from neuron j (or from network
input j )toneuron i .
The output g ( x , w ) of the network is a linear function of the parameters
of the last connection layer (connections that convey information from the N c
hidden neurons to the output neuron N c + 1), and it is a nonlinear function