Probabilistic and Statistical Methods - Information Fusion in Signal and Image Processing - page 66

The approach in principal component analysis, which projects each source of infor-
mation on the eigenvectors of the covariance matrix, is often used, in order to obtain
l new decorrelated sources, arranged in decreasing order of energy. Truncating the set
to keep only the first l ( l>l ) sources can often be done while preserving most of the
original set's energy.

However, in practice, this method quickly shows its limitations in image process-
ing, for example, because it cannot take into account complex dependences between
images or the spatial variations of dependences.

In order to express the information contributed by adding a new source of infor-
mation I l +1

, the preferred approach is that sug-
gested by Shannon, which relies on the concepts of information and entropy [KUL 59,
MAI 96]. Based on the joint probability of the first l sources p ( I 1 ,...,I l ) (estimated
most of the time by using frequencies of occurrence, for example, based on the multi-
dimensional histogram of gray levels in an image), the entropy (or mean information
per pixel in the case of images) of the first l sources is defined by:
H I 1 ,...,I l =

to an already known set { I 1 ,...,I l }

p I 1 ,...,I l log p I 1 ,...,I l ,

− [6.1]

and the entropy contributed by the ( l +1) th source is expressed, either depending on
the entropies, or depending on the probabilities, as:
H I l +1 |

I 1 ,...I l = H I 1 ,...,I l +1 − H I 1 ,...,I l

p I 1 ,...I l +1 log p I l +1 | [6.2]

I 1 ,...,I l .

= −

For two sources, we thus define redundancy 1 between them as:

R ( I 1 ,I 2 )= H ( I 1 )+ H ( I 2 ) − H ( I 1 ,I 2 ) , [6.3]

and the complementarity of the source I 2 with respect to I 1 , i.e. the mean quantity of
information that has to be added to I 2 in order to have I 1 :
C I 1 | I 2 = H I 1 | I 2 , [6.4]

which leads us to the following relation:
H I 1 = R I 1 ,I 2 + C I 1 | I 2 .

[6.5]

Analogous methods could be considered in a non-probabilistic framework, by rely-
ing, for example, on fuzzy entropy [LUC 72]. For the moment, the formalism is less
well developed in this direction.

1. This redundancy unfortunately cannot be generalized to more than two sources without
potentially losing its positivity property.