Neural Networks Approach - Computational Intelligence in Time Series Forecasting - page 108

3.5.3 Network Training Strategy
Network training is a process in which the network learns to recognize the patterns
inherent to the training signals. In network training for time series forecasting all
relevant characteristic features embedded in the training data that reflect the
autocorrelation structure of the time series should be revealed and learnt. The
training is usually carried out in off-line mode using an unconstrained nonlinear
minimization algorithm, most frequently a gradient descent method, for tuning the
interconnection weights of the network. The objective is to achieve the optimal
network behaviour across the training set.
Network learning can generally be executed in supervised mode (Hopfield
model) or in unsupervised mode (Kohonen model). For supervised learning the
network is provided by data examples that include the desired output. For
unsupervised learning the desired output values are not required because the
network finds the adequate output values itself.
The objective of training is to find the set of most suitable values of
interconnecting weights through their tuning during the network training. By doing
so, the network should still attain the highest generalization attribute . This,
however, can be aggravated if, instead of the global minimum, only a local
minimum has been found. So, particular precautions should be provided to avoid
pitting into one of the local minima. Such and similar issues seriously affect the
training success, so that some careful considerations are required when preparing
the experiment design for network training. This includes some decisions to be
made concerning the network initialization for training, selection of the appropriate
training algorithm, monitoring the training process using an appropriate
performance index, formulation of training stopping criteria, etc .
Network initialization is a decision that is to be made before the weights tuning
process starts. This is a difficult decision, because the training speed and the total
training time required are strongly influenced by this decision. To circumvent this,
various suggestions have been made, the most popular being that, in order to
prevent neuron saturation and other unpleasant phenomena, some small, randomly
distributed parameter values should initially be taken. However, setting all weights
initially at the same small value should be avoided because it could possibly
hamper the tuning process to start and/or to learn. This definitely does not hold for
unsupervised training, like it holds for training of a Kohonen layer of a
counterpropagation network, where the competition process take place. Here, the
unique value 1/ N is initially taken for all weights, N being the number of
network inputs. This is required because by starting the competition process it is
advantageous that all competitors have the same initial parameter values for every
training run.
Hebb (1949) has proposed the simplest training algorithm for neural networks,
known as the Hebb learning rule . A neurophysiologist himself, he enunciated the
learning principle of natural neurons: if two interconnected neurons at the same
time fire, then the strength (weight) of the synapse connecting them increases.
Extended to artificial neural networks, this principle states that the common weight