Biomedical Engineering Reference
In-Depth Information
the projection onto the projection space are being adapted at the same time. Another rule of thumb
is to adapt the bases at a slower rate than the projection, which means that the learning rate of the
first layer weights should be smaller than of the second layer weights (even after recognizing that the
errors are attenuated by passing through the hidden PE nonlinearity) and to train the network with
early stopping criterion (i.e., set aside 10% of the data for a cross-validation (CV) set, and stop train-
ing when the error in the CV increases. The error in the CV set should always be comparable to the
error in the training set, otherwise, overtraining occurred and generalization will be compromised.
Here, we present the training of a TDNN for the BMI data described in Section
3.1.1
.
The networks implemented consisted of one hidden layer with a hyperbolic tangent nonlinearity
(tanh(
βx
)). Because the dynamic range of the input data was large (0-21), the slope of the non-
linearity had to be carefully chosen. A beta value of 0.5 provided a sufficiently steep slope, which
with slow learning (over many epochs) nonlinearities would saturate to produce flat regions in the
trajectory. The slope was also shallow enough to prevent instantaneous jumps in firing rate from
saturating the nonlinearity and decreasing learning. The output layer of the network consisted of a
linear PE because the desired signal was hand position in space. The network described was trained
using static backpropagation. Weights were updated in online mode. Because the data have a large
variability, the weights needed to be updated at each time instance; batch updates tend to average
out the updates making learning difficult for this problem.
FIgURE 3.:
Learning curve for a BMI TDNN. Here, the
y
axis relates the MSE to the number
of training epochs. Included in the figure is the CV curve. Theoretical analysis indicates that training
should be stopped at the minima of the CV curve.
