Information Technology Reference
In-Depth Information
a b c
accuracy
0.9350 0.9100 0.8790
e
−
distance
0.9269 0.9256 0.9024
f
−
measure
0.8036 0.7761 0.6254
g
−
mean
0.9276 0.9379 0.9106
t
−
area
0.9276 0.9388 0.9115
Plot of ROC Points
Accuracy comparison
1
1
b
c
a
0.95
b
a
b
a
a
a
b
0.8
c
b
c
0.9
c
c
0.85
0.6
a
0.8
b
0.4
0.75
0.7
0.2
0.65
c
0
0.6
0
0.2
0.4
0.6
0.8
1
a
e
f
g
t
false positive fraction
measures
Fig. 1.
Accuracy measure results for AT systems
a
,
b
, and
c
.
Thus, the goal here is to maximize the expected value (EV) of each decision.
The EC of a non-parametric AT system or a parametric AT system operat-
ing at a given decision threshold is given by:
P
(
C
+)
· P
(
D
+
| C
+)
· B
(
D
+
|
C
+)+(1
− P
(
C
+))
· P
(
D−|C−
)
· B
(
D−|C−
)+
P
(
C
+)
· P
(
D−|C
+)
·
C
). Two consequences
are inferred directly from that equation. First, two AT systems will have the
same EV if:
TPF
2
−TPF
1
FPF
2
−FPF
1
(
D−|C
+) + (1
− P
(
C
+))
· P
(
D
+
| C−
)
· C
(
D
+
| C−
1
−P
(
C
+)
P
(
C
+)
×
B
(
D−|C−
)+
C
(
D
+
|C−
)
B
(
D
+
|C
+)+
C
(
D−|C
+)
. This equation de-
fines the slope of an
iso-performance
line [2]. Second, the slope that corre-
sponds to the optimal decision threshold
=
S
optimal
can be computed as follows
1
−P
(
C
+)
P
(
C
+)
×
B
(
D−|C−
)+
C
(
D
+
|C−
)
B
(
D
+
|C
+)+
C
(
D−|C
+)
.
The ultimate objective of intrusion detection is to develop robust systems
able to face imprecise environments where the operational costs of an IDS will
depend on the importance of the target's mission, the nature of possible future
attacks, and the level of hostility. We describe how to evaluate AT systems in
these environments in next subsection.
1.3 Alert Triage Evaluation in Imprecise Environments
These scenarios are useful for the evaluation of systems for real-world deployment
where misdetection costs not only will be unknown a priori but also will vary
over time. To decide whether an AT system outperforms others we will use a
robust and incremental method for the comparison of multiple detection systems
in imprecise and dynamic environments that has been proposed in [2]. This
method, named ROCCH (ROC Convex Hull) is a combination of ROC analysis,
