Environmental Engineering Reference
In-Depth Information
Figure 2. Diagram of Independent Component
Analysis (ICA) mixing and separating matrices
In the simplest ICA models - including Blind
Signal Separation (BSS) - the number of signals
is equal to the number of sources: the rotation
matrix is of full rank.
The Copula Approach
A
copula
is a multivariate distribution on mar-
ginal distribution functions—a distribution func-
tion on a
k
−dimensional cube—and holds the
dependency of the full joint distribution. In il-
lustration: take two random variables
X
~
F
,
X
~
F
.
1
X
2
X
1
2
Independent Component Analysis (ICA) can
be cast as a generalization of the PCA program
where more general versions of statistical inde-
pendence succeed covariation and thus uncorre-
latedness (Jutten and Herault [1991]). In both
versions the objective is the recovery of the linear
rotation
A
of the independent signals,
x
. The
difference is the characterization of statistical
independence or contrast function, and the im-
plicit or explicit distributional assumptions on the
inputs (See Cardoso [1993], Brunel et al. [2005]).
ICA extends independence beyond covariance.
While zeroed covariation is sufficient for inde-
pendence under the Gaussian assumption typi-
cally operant in PCA, when dependency is not
appropriately captured by the second moment,
covariance is an insufficient proxy for statistical
independence. For a simple example, take func-
tional dependency
x
A copula is a function that takes the 'grades'
as arguments—the pair
( ,
U V
are the 'grades' of
)
(
X X
1
,
2
--- and returns a joint distribution func-
)
tion
( ,
)
,
C U V
=
F
X X
,
1
2
with marginals
F F
X
, . In a simple illustration,
t h e G u m b e l - H o u g a r d c o p u l a —
C u v
X
1
2
( , )
= + − + −
(
u
v
1
)
(
1
u
)(
1
−
v
) *
e
−
θ
ln(
1
−
u
) ln(
1
−
v
)
θ
—is easily derived from the bivariate exponential
distribution:
H x y
−
x
−
y
− + +
(
x
y
θ
xy
)
( , )
= −
1
e
−
e
+
e
. Notice if
θ
θ = 0 , then
C u v
θ
( , )
= ...the independence
uv
copula.
The copula families of multivariate distribu-
tions, those where a candidate joint distribution
is evaluated on a set of univariate marginals, are
functions from I
k
to I . As densities -- full de-
rivatives -- copulas are the ratio of the joint den-
sity to the product of the univariate marginals (see
Nelsen [1999]). In this sense the copula represen-
tation captures the dependence within
x
: the
value of the copula is the proportion of dependence
to full independence. This proportion is maximal
when there is no gain to modelling a multivariate
2
, for example,
=
h x
=
x
(
)
i
j
j
x
i
= 0 . Here $Cov(x_i,x_j) = 0$ though
$x_i,x_j$ are completely statistically dependent.
ICA can be seen as PCA under a more general
contrast function, based on an alternate measure.
In PCA we seek the linear rotation that mini-
mizes covariance; in ICA we seek the rotation
that minimizes, for example: entropy, mutual
independence, higher order de-correlation, etc.
(Cardoso [1996]).
E
( )
