Information Technology Reference
In-Depth Information
8.2 Consistency of Learning Processes
8.2.1
Classical definition of learning consistency
In order to construct algorithms for learning from a limited number of
observations, we need an asymptotic theory (consistency is an asymptotic
concept). We describe the conceptual model for learning processes that are based
on the empirical risk minimization inductive principle. The goal of this part is to
describe necessary and sufficient conditions for the consistency of learning
processes that minimize the empirical risk.
Definition
8.1
The
Empirical
Risk
Minimization
Inductive
Principle
(Vapnik ,1995).
We say that the principle (method) of ERM is consistent for the set of
functions
) if the following
two sequences converge in probability to the same limit (see the schematic
Figure 8.l):
L
(
y,w
) and for the probability distribution function
F
(
y
R(w
l
)
Inf R(w)
R
emp
(w
l
)
Figure 8.1 Consistency of Learning Process
P
R
(
w
)
→
inf
R
(
w
)
(8.4)
l
l
∞
w
∈
P
R
(
w
)
→
inf
R
(
w
)
(8.5)
emp
l
l
∞
w
∈
In other words, the ERM method is consistent if it provides a sequence of
functions
= 1,2, ... , for which both expected risk and empirical risk
converge to the minimal possible value of risk. Equation (8.4) asserts that the
values of achieved risks converge to the best possibility, while Equation (8.5)
L
(
y,w
l
),
l
