Database Reference
In-Depth Information
Rather than using the basic implementation for the computation of the
change in weight, the Layer implementation stores the weight delta values
using a separate matrix:
double
dW[][];
double
dbW[];
double
m = 0.1;
public
MomentumLayer(
int
units,
int
inputs,
Activation fn,
boolean
bias) {
super
(units, inputs, fn, bias);
dW =
new double
[units][];
for
(
int
i=0;i<v.length;i++) dW[i] =
new
double
[inputs];
if
(bias)
dbW =
new double
[units];
}
Theupdate stepthen usesaweighted average ofthenew delta fortheweight
and the previous delta for the weight with the weight being chosen by a
constant value
m
:
@Override
public double
[] update(
double
[] o,
double
r) {
for
(
int
i=0;i<v.length;i++) {
if
(bW !=
null
) {
dbW[i] = (1-m)*r*err[i] + m*dbW[i];
bW[i] += dbW[i];
}
double
[] W = w[i];
for
(
int
j=0;j<W.length;j++) {
dW[i][j] = (1-m)*r*err[i]*o[j] + m*dW[i][j];
W[j] += dW[i][j];
}
}
return
v;
}
This allows weight changes in the same direction to proceed more rapidly,
while sudden changes in the direction of the weight have less of an effect. In
