Modelling of Intrusion Detection System Using Artificial Intelligence—Evaluation of Performance Measures - Complex System Modelling and Control Through Intelligent Soft Computations - page 316

3.1 Evaluation Metrics

An elementary concern in the development of classi cation models is the evalua-

tion of predictive accuracy (Guisan and Thuiller 2005 ; Barry and Elith 2006 ). The
quantitative evaluation of the model is important as it helps in determining the
ability of the model tom provide better solution for a speci c problem and also

assist in exploring the areas of model improvement. In the domain of binary pre-
dictions of anomaly and normal attacks, a confusion matrix (Table 1 ) known as
contingency table or error matrix (Swets 1988 ) that represents the performance
visualization of the predictive models of IDS that consists of two rows showing the
actual class and two columns showing the predicted class. The aim is to check
whether the system is confusing both classes. The IDSs are primarily distinguished
binary classes: anomaly class (malicious, threats or abnormal data) and normal class
(normal data points). Therefore, the proposed models generating normal-anomaly
predictions of intrusion detection system are typically assessed in Table 1 through
comparison of the predictions and developing a confusion matrix to predict the
number of true positive (TP), false positive (FP), false negative (FN) and true
negative (TN) cases. TP/TP+FN, is used as detection rate (DR) or sensitivity. It is
also termed as recall in information retrieval Overall accuracy is a simple measure
of accuracy that can be derived from the confusion matrix by calculating the
proportion of correct prediction. Sensitivity is the proportion of observed normal
attacks that are predicted as such, and therefore quanti es omission errors. Speci-

ficity is the proportion of observed anomaly attacks that are predicted as such, and
therefore quanti city are independent

of each other when compared across models. The most popular measure for the
accuracy of yes

es commission errors. Sensitivity and Speci

is kappa (Shao and Halpin 1995 ; Segurado
and Araujo 2004 ) which corrects the overall accuracy of model predictions by the
expected random accuracy. The kappa statistic ranges from 0 to 1, where 1 indicates
perfect agreement and values of zero indicate a performance no better than random
(Cohen 1960 ). The principle benefits of kappa are for its simplicity and the reason
that both commission and omission errors are accounted for in one parameter. In
this paper we also introduced another measure known as the true skill statistic
(TSS) for the performance of normal

- no predictions is Cohen '

anomaly classi er models, that still preserves

-

the advantages of kappa.
In the next section a detail experiment and analysis demonstrated the ef cacy of

the proposed MLP in the development of IDS system based on the above discussed
classi

cation evaluation metrics.

Table 1 Confusion matrix

Predicted class

Actual class Anomaly Normal

Anomaly TP FN

Normal FP TN