Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 113

Furthermore, v k m = i w mi x i = i w mi f ( v i ), therefore ( ∂v m ) / ( ∂v i ) k =

w mi f ( v i ).

Finally, one gets

=
m

δ k m w mi f ( v i )= f ( v i )
m

δ i δ k m w mi .

Thus, the quantities δ i can be computed recursively from the outputs to the
inputs of the network, hence the term backpropagation .
Once the gradients of the partial costs are computed, the gradient of the
total cost function is obtained by a simple summation.

Summary of Backpropagation

For each example k , the backpropagation algorithm for computing the gradi-
ent of the cost function requires two steps,

• A propagation phase, where the inputs corresponding to example k are

input to the network, and the potentials and outputs of all neurons are
computed,

A backpropagation phase, where all quantities δ i

are computed.

•

When those quantities are available, the gradients of the partial cost functions
are computed as ( ∂J k ) / ( ∂w ij ) k

= δ i x j , and the gradient of the total cost

function as ( ∂J ) / ( ∂w ij ) k = k ( ∂J k ) / ( ∂w ij ) k .
The backpropagation algorithm can be interpreted graphically by defining
the adjoint network of the network whose parameters must be estimated. This
approach is sometimes useful; it is discussed in Chap. 4 for the modeling of
dynamic systems.
Backpropagation was discussed here in the framework of the minimization
of the least squares cost function. It can be adapted to the minimization of
alternative cost functions, such as the cross entropy cost function, used for
classification.

Forward Computation of the Gradient of the Cost Function

One of the most persistent myths in the field of neural networks is the fol-
lowing: the invention of backpropagation made the development of neural
networks possible. Actually, it is definitely possible, albeit more computation-
ally demanding, to compute the gradient of the cost function in the forward
direction. That algorithm was extensively used for the estimation of the pa-
rameters of cascaded filters, long before backpropagation.
The forward algorithm proceeds as follows:

For a neuron m , which receives the quantity x j

• directly from input j of

the network or from neuron j ,