Search Based Software Engineering: Techniques, Taxonomy, Tutorial - Empirical Software Engineering and Verification - page 18

confidence level, then we are increasingly likely to commit a Type I error. This
situation is amusingly captured by an xkcd cartoon [73].
Sometimes, we find ourselves comparing, not vales of measurements, but the
success rates of searches. Comparison of success rates using inferential statis-
tics requires a categorical approach, since a search goal is either fulfilled or not.
For this Fisher's Exact test is a useful statistical measure. This is another non-
parametric test. For investigative of correlations, researchers use Spearman and
Pearson correlation analysis. These tests can be useful to explore the degree to
which increases in one factor are correlated to another, but it is important to
understand that correlations does not, of course, entail causality.

7 More Advanced Techniques

Much has been achieved in SBSE using only a single fitness function, a sim-
ple representation of the problem and a simple search technique (such as hill
climbing). It is recommended that, as a first exploration of SBSE, the first ex-
periments should concern a single fitness function, a simple representation and
a simple search technique. However, once results have been obtained and the
approach is believed to have potential, for example, it is found to outperform
random search, then it is natural to turn one's attention to more advanced tech-
niques and problem characterisations.
This section considers four exciting ways in which the initial set of results
can be developed, using more advanced techniques that may better model the
real world scenario and may also help to extend the range and type of results
obtained and the applicability of the overall SBSE approach for the Software
Engineering problem in hand.

7.1 Multiple Objectives

Though excellent results can be obtained with a single objective, many real world
Software Engineering problems are multiple objective problems. The objectives
that have to be optimised are often in competition with one another and may be
contradictory; we may find ourselves trying to balance the different optimisation
objectives of several different goals.
One approach to handle such scenarios is the use of Pareto optimal SBSE,
in which several optimisation objectives are combined, but without needing to
decide which take precedence over the others. This approach is described in more
detail elsewhere [48] and was first proposed as the 'best' way to handle multiple
objectives for all SBSE problems by Harman in 2007 [36]. Since then, there has
been a rapid uptake of Pareto optimal SBSE to requirements [27, 31, 84, 90, 113],
planning [5, 98], design [17, 88, 95], coding [9, 99], testing [33, 35, 47, 76, 90, 96,
107], and refactoring [52].
Suppose a problem is to be solved that has n fitness functions, f 1 ,...,f n
that take some vector of parameters x . Pareto optimality combines a set of
measurements, f i , into a single ordinal scale metric, F , as follows: