Modeling with Neural Networks: Principles and Model Design Methodology - Neural Networks: Methodology and Applications - page 198

if Z T Z is well conditioned. Otherwise, that computation will give values of
the leverages that are larger than 1, or negative.
A better solution consists in decomposing matrix Z as

Z = UWV T ,

where:

U is an ( N , q ) matrix such that U T U = I .

•

• W is a diagonal ( q,q ) matrix, whose diagonal terms called singular values

of Z , are positive or zero, and ranked in order of decreasing values.

• V is a ( q,q ) matrix, such that V T V = VV T = I .

That decomposition, known as singular value decomposition or SVD decompo-
sition, is accurate and robust, even if Z is ill-conditioned, or has rank smaller
than q (see [Press et al. 1992], and Chap. 3).
Thus, one has

Z T Z = VWU T UWV T = VW 2 V T ,

then

( Z T Z ) − 1 = VW − 2 V T .
That decomposition allows the direct computation of matrix ( Z T Z ) − 1 , the
elements of which can be written as

= q
k =1

Z T Z − 1
lj

V lk V jk
W 2
kk

.

After some algebra, one gets

q q

h kk = z k T Z T Z − 1 z k = Z kl Z kj Z T Z − 1

lj

,

l =1 j =1

and, finally

⎛ ⎞

2

q q

1
W ii

⎝ ⎠

h kk = Z kj V ji .

i =1 j =1

Thus, the leverages can be computed without resorting to the computation
of ( Z T Z ) − 1 , which is important in the case of ill-conditioned matrices. Since
the singular values are ranked in order of decreasing values, it is advantageous
to compute the leverages by varying i from q to 1, not from 1 to q .