Biomedical Engineering Reference
In-Depth Information
only by using a more elaborate model. The
minimum description length prin-
ciple
(184,185) enjoins us to pick the model that minimizes the description
length, and the
stochastic complexity
of the data is that minimized description-
length:
R
=
arg min
Cx
( , ,
R
2
,
)
[57]
MDL
R
R
=
min
Cx
( , ,
R
2 .
)
[58]
SC
R
Under not-too-onerous conditions on the underlying data-generating process and
the model class 2 (185, ch. 3), as we provide more data R
MDL
will converge on
the model in 2 that minimizes the generalization error, which here is just the
same as minimizing the Kullback-Leibler divergence from the true distribution.
25
Regarded as a principle of model selection, MDL has proved very success-
ful in many applications, even when dealing with quite intricate, hierarchically
layered model classes ((186) presents a nice recent application to a biomedical
complex system; see ยง3.4 for applications to state-space reconstruction.) It is
important to recognize, however, that most of this success comes from carefully
tuning the model-coding term
D
(R,2) so that models that do not generalize well
turn out to have long encodings. This is perfectly legitimate, but it relies on the
tact and judgment of the scientist, and often, in dealing with a complex system,
we have no idea, or at least no
good
idea, what generalizes and what does not. If
we were malicious, or short-sighted, we could always ensure that the particular
data we got have a stochastic complexity of just one bit.
26
The model that gives
us this complexity will then have absolutely horrible generalization properties.
27
Whatever its merits as a model selection method, stochastic complexity
does not make a good measure of the complexity of natural systems. There are
at least three reasons for this.
1.
The dependence on the model-encoding scheme, already dis-
cussed.
2.
The log-likelihood term,
L
(
x
,R) in
C
SC
can be decomposed into
two parts, one of which is related to the entropy rate of the
data-generating process, and so reflects its intrinsic unpredict-
ability. The other, however, indicates the degree to which even
the most accurate model in R is misspecified. Thus it reflects
our ineptness as modelers, rather than any characteristic of the
process.
3.
Finally, the stochastic complexity reflects the need to specify
some particular model, and to represent this specification.
