Knowledge Discovery and Data Mining in Medicine - Advanced Techniques in Knowledge Discovery and Data Mining - page 182

another new neuron. Eq. (7.1) is approximated by the following operational
measure:

& &

[] [ ] [] [ ]

WD n = γ WD n − 1 + ( − γ ) Met ( W m , W m − 1 ), 0 < γ < 1 . (7.4)

j W j W j j W

A neuron j should generate another neuron if

∆

∂ ε

ε = WD > θ , (7.5)

j j G

∂ WD

j

where is the overall system error, ε is the contribution of neuron j to the

ε

j

overall system error, WD is the WD of neuron j , and θ is a certain

j G

threshold value; ε determines the neuron generation.

j

7.2.2.2 Neuron Annihilation
To achieve an overall good system performance, we must remove unnecessary
elements from the network. Due to the competitive mechanisms between neurons
and the selection rule that makes the neurons keep their capability of finding their
correct positions in the network and removes misplaced neurons, the network
gradually forms an optimal structure in the development process. To be more
specific, we have the following two observations:
z If a neuron does not form the appropriate interconnections with other neurons,
it will die in the early development process.
z The neurons fight with each other because each neuron tries to stop other
neurons from taking exactly the same functional role in the network.

The following criteria for the neuron annihilation process were derived from
these observations. We kill the corresponding neuron if (A) the neuron is not a
functional element of the network or (B) the neuron is a redundant element of the
network. Criterion (A) can be identified by monitoring the output activity of the
neuron. As a measure for this criterion, we use the variance of the output activity
of neuron j :

2

VA = ( O − O )

j j j

, (7.6)

where

O is the output activation level for neuron

j and O is the average of

j

O over all training data. We defined the operational measure

VA as

j

[] [ ] []

2

VA n = γ VA n − 1 + ( − γ )( O − Act n ) , < γ < 1

j V j V j j V

, (7.7)

[] []

where VA j n is an estimate of VA at iteration n and Act j n is the

j

operational measure of the average output activity O at iteration n for neuron

j

j :

[] [ ] []

Act n = γ Act n − 1 + ( − γ ) O n , 0 < γ < 1

j a j a j a

. (7.8)

VA is very closely related to the information content of the output activities of
neuron

j

j .