Biomedical Engineering Reference
In-Depth Information
this PDF directly because we do not know which regime is active at which times, but we can observe
its marginal distribution
K
∑
p x
( )
χ
n
(
(
-
1
)
)
=
p k x
( ,
χ
n
1
(
(
-
)
)
=
k
=
1
K
∑
K
∑
P k
χ
n
1
(
(
-
)
)
p x
( )
χ
n
1
⋅
(
(
-
)
k
,
)
=
g
k
χ
n
(
(
-
1
)
)
p x
( )
χ
n
1
⋅
(
(
-
)
k
,
)
(3.32)
k
=
1
k
=
1
where
g
k
χ
n
1
(
≡
can be regarded as the a priori probability of the
k
th regime. If we
can find predictors for each of the subprocesses, up to a random term
ε
k
,
(
(
-
)
)
P k
χ
n
1
(
-
)
x
( )
=
f
χ
n
1
(
(
-
)
θ
k
;
)
ε
k
( )
+
k
=
1…
K
(3.33)
then we can evaluate the conditional PDF in terms of the innovations
p x
( )
χ
n
1
(
(
-
)
k
,
)
=
p
ε
k
ε
k
( )
(
)
(3.34)
The residuals are usually modeled as a Gaussian distribution, which for a 1D time series
becomes
~
−
1
(
x
(
n
)
x
(
n
))
2
(3.35)
k
p
k
(
ε
)
=
exp
ε
k
2
2
σ
2
πσ
2
k
k
~
where we have
χ
−≡
defined to be
k
th predictor's estimate of the next value of the
time series and σ
k
, a “nuisance parameter,” is the variance of the kth predictor. Taking the expected
value of both sides of Eq 3.33 gives the best MMSE prediction of the next value of the time series
x
(
n
)
f
(
(
n
1
);
θ
)
k
k
K
K
∑
∑
~
~
x
n
)
≡
E
[
x
(
n
)
|
χ
(
n
−
1
)]
=
g
(
χ
(
n
−
1
))
⋅
E
(
x
(
n
)
|
χ
(
n
−
1
),
k
]
=
g
(
χ
(
n
−
1
))
x
(
n
)
(3.36)
k
k
k
k
=
1
k
=
1
which is a weighted sum of outputs of the individual predictors. We can regard this as the total
system output. In the mixture-of-experts algorithm of Jordan and Jacobs [
50
], this mapping is pro-
vided by an adaptable function called the “gate.” The particular variation of the algorithm we will
examine is Weigend's [
51
] nonlinear gated experts, which implements the gate using a single MLP.
This architecture is shown in Figure
3.16
, and note that the individual predictors, called “experts” in
this context, and the gates all see the same input. In other implementations, the gate can be trained
from the outputs (called therefore output gating).
We now turn to the question of how to train the gate and experts simultaneously, which is
the great appeal of this architecutre. In the following development, for ease of reading, we leave out
