Computational Intelligence Techniques - Computational Intelligence in Biomedical Engineering - page 85

where u p is the weighted sum input for node p . The contribution of u i at the
node i to the error of a training pattern can be measured by computing the
error derivative written as

δ i = ∂ E k
∂ u i

= ∂ E k
∂ c i

∂ c i
∂ u i

(3.10)

For output nodes this can be written as

c i ) f i δ i = − ( y i − (3.11)

where f i is the derivative of the activation function. This will vary depending
on the activation function where, for example, the sigmoid function gives a
particularly nice derivative, that is, f i = f ( u i )(1 − f ( u i )). The tanh function

is equally elegant with f i =1 f 2 ( u i ).

For nodes in the hidden layers, note that the error at the outputs is influ-
enced only by hidden nodes i , which connect to the output node p . Hence for
a hidden node i ,wehave
δ i = ∂ E k
∂ u i

−

=
p

∂ E k
∂ u p

∂ u p
∂ u i

(3.12)

But the first factor is δ p of the output node p and so
δ i =
p

δ p ∂ u p
∂ u i

(3.13)

The second factor is obtained by observing that node i connects directly to
node p and so (∂ u p ) / (∂ u i )= f i w pi . This gives
δ i = f i
p

w pi δ p (3.14)

where δ i is now the weighted sum of δ p values from the nodes p , which are con-
nected to it (see Figure 3.8). Since the output values of δ p must be calculated

i +1

w i +1, i

Slope of
nonlinearity

i

P

∑

w pi

w Ni

N

i = f ′ ( u i ) ∑ w pi p

p > i

FIGURE 3.8
Backpropagation structure. Node deltas are calculated in the backward
direction starting from the output layer.