Neural Networks Approach - Computational Intelligence in Time Series Forecasting - page 120

In practice, to apply the above procedures, the second derivative (Buntine and
Weigend, 1994) of the inverse of Hessian matrix (Hassibi et al .,1992) has to be
calculated anew for every weight to be eliminated. Stahlberger and Riedmiller
(1996) proposed a fast network pruning method, called Uni-OBS, that still relies on
the optimal brain surgeon procedure but it requires only a single calculation of the
inverse Hessian matrix to eliminate a group of weights. This certainly simplifies
the calculation of net pruning. For accelerated calculations of matrix
multiplication, some fast computational algorithms are required or some algebraic
transformations that also accelerate the calculation process. An amendment of the
Uni-OBS method, called G-OBS ( generalised optimal brain surgeon ), can
simultaneously eliminate, say m , weights in one step with slight increase in error
given as

1
2

G E G wHw

T

G ,

The related elimination condition is given by

) T

( wwS G

,

m

S being the selection matrix that determines the m weights to be removed
simultaneously. Using the above weights elimination conditions and the
corresponding Lagrange method, we get for the resulting error the relation

1

) T T

1

G wHSS ( SSw

and

1

G EwSS

T

(

T

SSw )

1 T

.

2

For acceleration of the pruning process, Levin et al. (1994) proposed a method for
elimination of excess weights.
Another way was followed by Jollife (1986). To improve the network
generalization capability, he used the method of principal component analysis .
This is a valuable mathematical tool for reducing a system's dimensionality by
eliminating it's redundant variables. This method transforms the variables to a
basis in which the system covariance is diagonal and the projection is in the low
variance directions. To detect the variables that have a low significant influence on
the error function, a salience measure is used, which demonstrates the
relationships between the proposed methods and the optimal damage and optimal
surgeon procedures of network pruning. The pruning consists in removing the
eigen-nodes with low saliency to reduce the effective number of network
parameters. In contrast to the optimal brain damage and optimal brain surgeon
procedures, which reduce the rank by eliminating actual weights, the proposed