Neuro-fuzzy Approach - Computational Intelligence in Time Series Forecasting - page 240

variance parameters

V of the Gaussian membership functions; therefore,
Equations (6.42a) - (6.42c) delivered the corresponding transpositions of the
Jacobian matrices and the Jacobian matrices themselves for the Takagi-Sugeno-
type multi-input, multi-output neuro-fuzzy network's free parameter and gave the
Levenberg-Marquardt algorithm better convergence in most experiments.

c and

l l

6.4.2.4 Adaptive Learning Rate and Oscillation Control
The proposed backpropagation training algorithm and the Levenberg-Marquardt
training algorithm, both with the modified error index extension as performance
function and with the added small momentum term, have proven to be very
efficient, and faster in training the Takagi-Sugeno-type neuro-fuzzy networks than
the standard back-propagation algorithm. But still, the performance function of the
network (if left without any proper care) is not always guaranteed to reduce, in
every epoch, towards the desired error goal. As a consequence, the training can
proceed in the opposite direction, giving rise to a continuous increase of
performance function or to its oscillation. This prolongs the training time or makes
the training impossible. To avoid this, three sets of adjustable parameters are
recommended to be stored for the backpropagation algorithm and two sets for the
Levenberg-Marquardt algorithm. The stored sets are then used in the following
way.
In the case of the backpropagation algorithm, if two consecutive new sets of
adjustable parameters reduce the network performance function, then in the
following epochs the same sets are used and the learning rate in the next step is
increased slightly by a factor of 1.1. In the opposite case, i.e . if the performance
function with the new sets of parameters tends to increase beyond a given limit -
say WF ( wildness factor of oscillation) times the current value of the performance
function - then the new sets are discarded and training proceeds with the old sets
of adjustable parameters. Thereafter, a new direction of training is sought with the
old sets of parameters and with lower values of the learning rate parameter, e.g . 0.8
or 0.9 times the old learning rate.
In the case of the Levenberg-Marquardt algorithm, if the following epoch
reduces the value of the performance function, then the training proceeds with a
new set of parameters and the P value is reduced by a preassigned factor

1 dec P .
In the opposite case, i.e. if the next epoch tends to increase this performance value
beyond the given limits ( WF times of current value of performance function) or
remains the same, then the P value is increased by another preassigned factor
( in P ) but the new set of adjustable parameters is discarded and training proceeds
with the old set of parameters. In this way, in every epoch the value of the
performance function is either decreased steadily or at least maintained within the
given limit values.