Biomedical Engineering Reference
In-Depth Information
Remarks:
More insights can be obtained through the SVD,
s
s
�
ˆ
(4.13)
W
=
P
diag
1
, ...,
r
,
0
, ... ,
0
Q y
T
TR
2
2
s
+
λ
s
+
λ
1
r
Comparing it with the pseudo-inverse solution
�
ˆ
(4.14)
W P
=
diag
(
s
−
1
, ... ,
s
−
1
,
0
, ..., )
0
Q y
T
1
r
we see that the Tikhonov regularization modifies the singular values through the following regu-
larization function:
(4.15)
2
2
λ
H x
TR
( )
=
x
/(
x
+
)
Note that
H
TR
( )®1
when
x
is small. In this sense, the Tik-
honov regularization smoothly filters out the singular components that are small (relative to λ). This
viewpoint actually brings new understanding into all kinds of regularization techniques as will be
described in this chapter.
TR
( )®1
when
x
is large and
H
x
x
4.1.1 Ridge Regression and weight decay
When building linear models for BMI with neuronal activity that is highly variable with a large
dynamic range of bin counts, the condition number of an input correlation matrix may be relatively
large as shown in Chapter
3
. In this situation, the performance of the model for reconstructing the
movement trajectories from neural activity is highly variable because of the extra degrees of freedom
that are not constrained. To reduce the condition number, we can, according to (
4.12
), add an iden-
tity matrix multiplied by a white noise variance to the correlation matrix, which is known as ridge
regression (RR) in statistics [
15
] when LS are being solved as in the case of the Wiener solution
We can also use a similar regularizer in the iterative updates based on the least mean squares
(LMS), or normalized LMS (NLMS) algorithm, which is called weight decay [
16
]. Rewriting
(
4.11
), the criterion function of the regularized solution is
2
d
(4.16)
J
(
w
)
=
E
[
e
]
+
w
where the additional term
δ
||
w
||
2
smoothes the cost function. The instantaneous gradient that cor-
responds to this equation can be written as
ˆ
(4.17)
(
n
+ = + ∇ −
1
)
( )
n
η ζ δ
( )
n
( )
n
w
w
w
