Introduction to Combinatorial Testing - Automatic Generation of Combinatorial Test Data - page 2

•

desktop, which can be manufactured by HP or Lenovo or Dell;

•

operating system, which can be Linux or Microsoft Windows;

• web browser, which can be Microsoft IE or Firefox;

• database management system, which can be Oracle or MySQL;

• ···

For another example, suppose we are performing unit testing, and the function to
be tested computes the product of two complex numbers: ( a 1 + b 1 i ) × ( a 2 + b 2 i ) .

So there are four parameters: a 1 , b 2 . For each parameter, the tester can select

several representative values (like Negative , zero , Positive ).
To test such a system, we may try all combinations of values and see if there is
any error. But the cost might be too high, because the number of test cases increases
exponentially when the number of parameters increases. Fortunately, experiences
indicate that many systems exhibit erroneous behaviors due to the combination of
just a few parameters' values. Thus, it is not necessary to check the combination of
all parameters.
Wallace andKuhn [ 41 ] analyzed software-related failures of somemedical devices
that led to recalls by the manufacturers. They found that many flaws could have been
detected by testing all pairs of parameter settings. The applications are typically
small to medium sized embedded systems.
Kuhn et al. [ 26 ] analyzed 329 error reports generated during the development and
integration testing of a large distributed system developed at NASA Goddard Space
Flight Center. The application is a data management system that gathers and receives
large quantities of scientific data. They found that the number of conditions required
to trigger a failure is at most six.
The above empirical studies show that software failures in many domains were
caused by combinations of relatively few conditions. This motivates the use of com-
binatorial testing (CT), whose goal is to cover as many combinations of parameters
or conditions as possible, using a relatively small test suite.
During the past 20 years or so, CT has been applied in many areas.
The AETG system [ 9 ] has been used in Bellcore for screen testing, interoperability
testing, and protocol conformance testing. (Screen testing means testing the software
that checks the user's inputs for consistency and validity.)
Cohen et al. [ 10 ] describes the application of CT to 10 Unix commands:
basename , cb , comm , crypt , sleep , sort , touch , tty , uniq , and wc .In
particular, they discussed several different ways to model the sort command.
Dunietz et al. [ 16 ] applied CT to an operations support system, which supports
the maintenance operations to the AT&T network.
Smith et al. [ 37 ] described how to validate theRemoteAgent Experiment (RAX) of
the Deep Space One (DS1) mission. The authors advocate a combination of scenario-
based testing and model-based validation. They used orthogonal arrays when design-
ing test suites. (For an introduction to orthogonal arrays, see Sect. 1.3.2 ).
Ryu et al. [ 34 ] applied CT to testing the data part of the Transmission Control
Protocol (TCP). Some typical discrete values were selected for some significant data

b 1 , a 2 ,