Database Reference
In-Depth Information
For simplicity, this implementation uses the same activation function for all
units.Itisnotrequiredthatallunitsinalayerusethesamefunction,butitis
oftenthecase.However,anyfunctionusedmusthaveaderivativetobeused
in thebackpropagation training algorithm discussed in thenext section. The
abstract
Activation
class defines methods for both the function and its
derivative:
public abstract class Activation {
public abstract double f(double x);
public abstract double df(double fx);
}
Two commonly used activation functions are the logistic function (also
called the sigmoid function) and the hyperbolic tangent. The logistic
function is implemented with a constant
k
that is used to control the rate of
change from positive to negative. A sufficiently large value for
k
of, say, 100,
is usually sufficient to approximate the Heaviside step function, which is not
ordinarily usable in the backpropagation algorithm:
public static
Activation logit(
final double
k) {
return new
Activation() {
@Override
public double
f(
double
x) {
return
1.0/(1.0 + Math.
exp
(-k*x));
}
@Override
public double
df(
double
fx) {
return
k*fx*(1.0-fx);
}
};
}
public static
Activation sigmoid(
double
k) {
return
logit
(k);
}
public static
Activation logit() {
return
logit
(1.0);
}
public static
Activation sigmoid() {