Biomedical Engineering Reference
In-Depth Information
is also called information divergence, information gain, or Kullback-Leibler diver-
gence. In MATLAB KL divergence between distributions can be computed using
function kldiv . This function can be downloaded from the MATLAB Central file
exchange http://www.mathworks.com/matlabcentral/fileexchange/ .
Mutual information is based on the concept of the entropy of information (see
Sect. 2.3.1.2). For a pair of random variables x and y the mutual information (MI)
between them is defined as:
p ij
p i p j
= p ij log
MI xy
(3.42)
where p ij is the joint probability that x
y j . This measure essentially tells
how much extra information one gets on one signal by knowing the outcomes of the
other one. If there is no relation between both signals MI xy is equal to 0. Otherwise,
MI xy will be positive attaining a maximal value of I x (self information of channel x )
for identical signals (Sect. 2.3.1.2).
MI is a symmetric measure and it doesn't give any information about directional-
ity. It is possible to get a notion of direction by introducing a time lag in the definition
of MI xy . However, it is transfer entropy which is frequently applied as a non-linear
directionality measure.
The transfer entropy was introduced by [Schreiber, 2000], who used the idea of
finite-order Markov processes to quantify causal information transfer between sys-
tems I and J evolving in time. Assuming that the system under study can be approxi-
mated by a stationary Markov process of order k , the transition probabilities describ-
ing the evolution of the system are: p
=
x i and y
=
(
|
,...,
)
i n + 1
i n
i n k + 1
. If two processes I and J are
i ( k )
i ( k )
j ( l )
independent, then the generalized Markov property p
(
i n + 1
|
)=
p
(
i n + 1
, |
,
)
n
n
n
holds, where i ( k )
i n k + 1 and j ( l )
is the number of con-
ditioning states from process I and J , respectively. Schreiber proposed to use the
Kullback- Leibler entropy (equation 3.40) to quantify the deviation of the transition
probabilities from the generalized Markov property. This resulted in the definition of
transfer entropy (TE):
=
i n
,...,
=(
j n
,...,
j n l + 1 )
n
n
i ( k )
j ( l )
p
(
i n + 1
|
,
)
= ( p ( i n + 1 , i ( k )
j ( l )
n
n
TE J I
,
)
log
)
(3.43)
n
n
i ( k )
(
|
)
p
i n + 1
n
The TE can be understood as the excess amount of bits that must be used to encode
the information of the state of the process by erroneously assuming that the actual
transition probability distribution function is p
i ( k )
j ( l )
.
MI and TE quantify the statistical dependence between two time series, with no
assumption about their generation processes, which can be linear or non-linear. De-
spite the apparent simplicity of the estimators, the practical calculation of MI or TE
from experimental signals is not an easy task. In practice estimation of the proba-
bilities involves obtaining the histograms of the series of outcomes and finding, say,
p i as the ratio between the number of samples in the i -th bin of the histogram and
the total number of samples. In order to get an accurate estimate of this measure by
(
i n + 1
|
)
instead of p
(
i n + 1
|
)
n
n
 
Search WWH ::




Custom Search