Image Processing Reference
The approach in principal component analysis, which projects each source of infor-
mation on the eigenvectors of the covariance matrix, is often used, in order to obtain
l new decorrelated sources, arranged in decreasing order of energy. Truncating the set
to keep only the first l ( l>l ) sources can often be done while preserving most of the
original set's energy.
However, in practice, this method quickly shows its limitations in image process-
ing, for example, because it cannot take into account complex dependences between
images or the spatial variations of dependences.
In order to express the information contributed by adding a new source of infor-
mation I l +1
, the preferred approach is that sug-
gested by Shannon, which relies on the concepts of information and entropy [KUL 59,
MAI 96]. Based on the joint probability of the first l sources p ( I 1 ,...,I l ) (estimated
most of the time by using frequencies of occurrence, for example, based on the multi-
dimensional histogram of gray levels in an image), the entropy (or mean information
per pixel in the case of images) of the first l sources is defined by:
H I 1 ,...,I l =
to an already known set
I 1 ,...,I l }
p I 1 ,...,I l log p I 1 ,...,I l ,
and the entropy contributed by the ( l +1) th source is expressed, either depending on
the entropies, or depending on the probabilities, as:
H I l +1 |
I 1 ,...I l = H I 1 ,...,I l +1 −
H I 1 ,...,I l
p I 1 ,...I l +1 log p I l +1 |
I 1 ,...,I l .
For two sources, we thus define redundancy 1 between them as:
R ( I 1 ,I 2 )= H ( I 1 )+ H ( I 2 )
H ( I 1 ,I 2 ) ,
and the complementarity of the source I 2 with respect to I 1 , i.e. the mean quantity of
information that has to be added to I 2 in order to have I 1 :
C I 1 |
I 2 = H I 1 |
I 2 ,
which leads us to the following relation:
H I 1 = R I 1 ,I 2 + C I 1 |
I 2 .
Analogous methods could be considered in a non-probabilistic framework, by rely-
ing, for example, on fuzzy entropy [LUC 72]. For the moment, the formalism is less
well developed in this direction.
1. This redundancy unfortunately cannot be generalized to more than two sources without
potentially losing its positivity property.