Biomedical Engineering Reference
In-Depth Information
the number of symbols in the input (167). This is an absolutely horrible idea;
even under the circumstances under which it gives a consistent estimate of the
entropy rate, it converges much more slowly, and runs more slowly, than em-
ploying either of the two techniques just mentioned (168,169). 20
7.2. Applications of Information Theory
Beyond its original home in communications engineering, information the-
ory has found a multitude of applications in statistics (159,160) and learning
theory (144,170). Scientifically, it is very natural to consider some biological
systems as communications channels, and so analyze their information content;
this has been particularly successful for biopolymer sequences (171) and espe-
cially for neural systems, where the analysis of neural codes depends vitally on
information theory (172,173). However, there is nothing prohibiting the applica-
tion of information theory to systems that are not designed to function as com-
munications devices; the concepts involved require only well-defined
probability distributions. For instance, in nonlinear dynamics (174,175) informa-
tion-theoretic notions are very important in characterizing different kinds of
dynamical system (see also ยง3.6). Even more closely tied to complex systems
science is the literature on "physics and information" or "physics and computa-
tion," which investigates the relationships between the mechanical principles of
physics and information theory, e.g., Landauer's principle, that erasing (but not
storing) a bit of information at temperature T produces k B T ln 2 joules of heat,
where k B is Boltzmann's constant.
8.
COMPLEXITY MEASURES
We have already given some thought to complexity, both in our initial
rough definition of "complex system" and in our consideration of machine learn-
ing and Occam's Razor. In the latter, we saw that the relevant sense
of"complexity" has to do with families of models: a model class is complex if it
requires large amounts of data to reliably find the best model in the class. On the
other hand, we initially said that a complex system is one with many highly
variable, strongly interdependent parts. Here, we will consider various proposals
for putting some mathematical spine into that notion of a system's complexity,
as well as the relationship to the notion of complexity of learning.
Most measures of complexity for systems formalize the intuition that some-
thing is complex if it is difficult to describe adequately. The first mathematical
theory based on this idea was proposed by Kolmogorov; while it is not good for
analyzing empirical complex systems, it was very important historically, and
makes a good point of entry into the field.
Search WWH ::




Custom Search