Security Oracle Based on Tree Kernel Methods - Trustworthy Eternal Systems via Evolving, Software Data and Knowledge - page 37

4 Empirical Results
Identification of candidate vulnerabilities has been performed by using Pixy [10],
a publicly available tool for taint analysis of PHP code, which reports a list of
candidate vulnerabilities derived from the source code of the application un-
der test. A vulnerability is represented by a sequence of target branches to
be executed, our vulnerability coverage criterion. A tool developed in previ-
ous works [1,2] is resorted to automatically generate an initial set of test cases
that satisfies the coverage criteria.
The experimentation has been conducted using SVM-light-TK 2 version 1.5
as kernel machine. This tool extends SVM-light tool 3 with kernel support by
implementing different kernel methods. The ones selected for the experiment are
listed in the following:
- Standard (Tree) Kernel (SK) [5],
- Sub Tree Kernel (STK) [19],
- Subset Tree Kernel (SSTK) [6],
- Subset Tree Kernel (SSTK) with bag-of-words (BOW) feature [21],
- Partial Tree Kernel (PTK) [14],
- Partial Tree Kernel with no leaves (uPTK) [15].
4.1 Prototype Results
To understand the applicability of the proposed approach in a small and control-
lable context, we first tested it on a mock-up case study, a simple web application
from which the running example of Figure 1 has been taken. The case study con-
sists of a single PHP script of 37 lines of code which represents a typical pattern
of a dynamic web page. It implements two different functionalities, according to
the value of an input parameter (generating a table or a sequence of links). This
script contains two XSS vulnerabilities.
A total of 3,470 test cases have been generated using previously developed
tools [1,2]. These tests have been manually filtered to remove false positives.
Among the remaining ones, we selected 460 safe executions and 46 code injection
attacks, to respect a 1:10 proportion among the two classes 4 . This corpus of data
has been randomly split into two parts, 50% for training and 50% for assessment.
While splitting data, we took care of dividing attacks uniformly between the two
parts.
Tuning of cost-factor value has been achieved with the following procedure.
Initially, only the 80% of the training data (202 test cases, training set) has
been used to build a preliminary model, while the remaining 20% (51 test cases,
tuning set) has been used to tune the cost-factor. Then, the initial model has
been adopted to classify the tuning set by iteratively changing the cost-factor
value from 1 to 50. We selected the optimal cost-factor value as the one that
2 http://disi.unitn.it/moschitti/Tree-Kernel.htm
3 http://www.joachims.org/
4 Generating attacks is usually harder than generating safe tests.