Computational Intelligence Techniques - Computational Intelligence in Biomedical Engineering - page 83

Input layer
Layer 0

Hidden layer
Layer 1

Output layer
Layer 2

FIGURE 3.7
A double-layer MLP network.

then the learning algorithm would not converge causing problems in further
applications. When these perceptrons are connected into a network, however,
they could form more complex separating boundaries (Minsky and Papert,
1988/1969). This architecture is the multilayer perceptron (MLP) and use of
the backpropagation algorithm (see next section) to train them has made
them one of the most popular and powerful neural network architectures. The
MLP architecture shown in Figure 3.7 consists of a single layer of inputs con-
nected to a layer of hidden units, which in turn are connected to the output
layer. This network is “fully connected” because each node in every layer is
connected to every node in the next layer. It is also possible to have a net-
work that is not fully connected. In general, an L -layer MLP network consists
of a single input layer, L 1 hidden layers and one output layer. There has

been some uncertainty concerning how to count the layers in an MLP net-
work (Reed and Marks, 1999), but it is now generally accepted that the input
layer is not included because it does not do any processing.
The number of hidden layers determines the complexity of the surface
boundary, for example, a single layer is only capable of producing linear sep-
arating planes whereas more layers can produce arbitrary separating bound-
aries depending on the number of nodes in the hidden layers. The sigmoid
activation function is also known to produce smooth separating boundaries
rather than just piecewise planar boundaries. Several design issues arise from
using the MLP structure that can only be empirically answered. These include
issues such as the number of hidden layers and nodes per layer to use, the node
interconnection topology, and the optimal training data size.

−

3.2.3 The Back-Propagation Algorithm
When the MLP architecture was introduced, an immediate problem that arose
was training it. How could the weights in the hidden layers be adjusted if one