Database Reference
In-Depth Information
For simplicity, this implementation uses the same activation function for all
units.Itisnotrequiredthatallunitsinalayerusethesamefunction,butitis
oftenthecase.However,anyfunctionusedmusthaveaderivativetobeused
in thebackpropagation training algorithm discussed in thenext section. The
abstract Activation class defines methods for both the function and its
derivative:
public abstract class Activation {
public abstract double f(double x);
public abstract double df(double fx);
}
Two commonly used activation functions are the logistic function (also
called the sigmoid function) and the hyperbolic tangent. The logistic
function is implemented with a constant k that is used to control the rate of
change from positive to negative. A sufficiently large value for k of, say, 100,
is usually sufficient to approximate the Heaviside step function, which is not
ordinarily usable in the backpropagation algorithm:
public static Activation logit( final double k) {
return new Activation() {
@Override
public double f( double x) {
return 1.0/(1.0 + Math. exp (-k*x));
}
@Override
public double df( double fx) {
return k*fx*(1.0-fx);
}
};
}
public static Activation sigmoid( double k) {
return logit (k);
}
public static Activation logit() {
return logit (1.0);
}
public static Activation sigmoid() {
Search WWH ::




Custom Search