Digital Signal Processing Reference
In-Depth Information
3
Information Theory and Principal Component
Analysis
In this chapter, we introduce algorithms for data analysis based on
statistical quantities. This probabilistic approach to explorative data
analysis has become an important branch in machine learning with many
applications in life sciences.
We first give a short, somewhat technical review of necessary con-
cepts from probability and estimation theory. We then introduce some
key elements from information theory, such as entropy and mutual in-
formation. As a first data analysis method, we finish this chapter by
discussing an important and often used preprocessing technique, princi-
pal component analysis.
3.1
Probability Theory
In this section we summarize some important facts from probability
theory which are needed later. The basic measure theory required for
the probability theoretic part can be found in many topics, such as [22].
Random Functions
In this section we follow the first chapter of [23]. We give only proofs
that are not in [244].
Definition 3.1: A
probability space
(Ω
,
A
,P
) consists of a set Ω, a
σ
-algebra
A
on Ω, and a measure
P
called
probability measure
on
A
with
P
(Ω) = 1.
While this may sound confusing, the intuitive notion is very simple:
For some subsets of our space Ω, we specify how probable they are.
Clearly, we want intersections and unions also to have probabilities, and
this (in addition to some technicality with respect to infinite unions) is
what is implied by the
σ
-algebra.
Elements of
A
are called events, and
P
(
A
) is called the
probability
of the event A
.Bydefinitionwehave
0
≤
P
(
A
)
≤
1
.
Search WWH ::
Custom Search