Information Technology Reference
In-Depth Information
w ( m 1)
k
+ η ∂ V R 2
∂w k
w ( m )
k
= w ( m− 1)
k
.
(3.11)
V R 2 .
4. With the new parameter vector compute the updated classifier outputs
y i = ϕ ( x i ; w ) and go to step 1 if some stopping condition is not met.
One may, of course, also apply gradient descent to
Note that the gradient descent algorithm for EE risks has a O ( n 2 ) complexity,
which is a disadvantage relatively to MSE and CE risks with O ( n ) complexity.
3.1.3 Fat Estimation of Error PDF
We open the present section with two simple examples of PDFs being itera-
tively driven to the theoretical MEE configuration. In spite of their simplicity
these examples serve well as forerunners of things to come.
Example 3.1. Consider the parametric PDF family
f ( x ; d, σ )= 1
2 ( g ( x ;
d, σ )+ g ( x ; d, σ )) ,
(3.12)
) is the Gaussian PDF with parameter vector w =[ ] T .Wewant
to study the entropy minimization of the family. We are namely interested, in
the present and following companion example, to interpret the PDF family
as an error PDF expressed as:
where g (
·
f E ( e )= 1
2 ( f E|− 1 ( e )+ f E| 1 ( e )) = 1
2 ( g ( e ;
d, σ )+ g ( e ; d, σ )) .
(3.13)
The class-conditional PDFs are equal-variance Gaussian PDFs and the priors
are equal ( p = q =1 / 2).
Note that this f E ( e ) corresponds to a classifier whose codomain is not a
bounded interval as in 2.2.1. Nevertheless, the requirement of driving f E ( e )
inside [
1 , 1] in order to reach a min P e =0configuration (if possible) still
holds. Driving f E ( e ) inside [
1 , 1] means decreasing both d and σ in such a
way that the f E ( e ) area outside [
1 , 1] is neglectable, and we are therefore
arbitrarily close to the min P e =0configuration. Asymptotically, w would
converge to [0 0] T and f E ( e ) towards a single Dirac- δ function at the origin.
We take the Rényi's quadratic entropy as risk functional (greatly facilitat-
ing the computations in comparison with Shannon entropy):
ln +
−∞
ln +
−∞
1
4 ( f E|− 1 ( e )+ f E| 1 ( e )) 2 de ; (3.14)
f E ( e ) de =
H R 2 ( E )=
 
Search WWH ::




Custom Search