Introduction to Combinatorial Testing - Automatic Generation of Combinatorial Test Data - page 4

•

a flexible manufacturing system which involves a conveyor, a camera, a machine
vision system and a robotic arm subsystem. It picks up metal sheets of different
shapes, colors, and sizes, and puts them into appropriate bins.

• an audio amplifier which has two input jacks, two volume controls, two toggle

switches, and one three-way selection switch.

• a password diagnoser for an online banking system, which verifies that the user

passwords conform to good security practices. For example, the length of a pass-
word should be at least eight (characters); a password should have at least one
upper-case character, one numeric character and one special character.

Calvagna et al. [ 5 ] described how to perform conformance testing of a Java Card
static verifier. They designed and implemented a combinatorial interaction based
testing approach to the validation of the verifier component of Java Card virtual
machines (JCVM), so as to check the degree of conformance to the verifier's design
specifications.
Garn and Simos [ 17 ] presented a new testing framework called ERIS, to test the
Linux system call APIs. The input space of the Linux system calls is modeled in
terms of CT.
Vilkomir and Amstutz [ 40 ] applied the combinatorial approaches to the testing of
mobile apps. The main goal is to reveal device-specific faults, and reduce the costs
of testing. This is achieved by an algorithm for optimal selection of mobile devices.

1.3 Related Combinatorial Designs

Given a model of the SUT (with some parameters and the corresponding values), how
can we come up with a small test suite that covers many combinations of parameters?
That is a challenging problem. Researchers and engineers have tried to borrow some
ideas and knowledge from the design of experiments.
This section describes some combinatorial designs [ 15 ] (including Latin squares,
orthogonal arrays, and covering arrays) that are commonly used in experimental
design. During the past 30 years, they have also been applied to software testing.
They serve as abstract representations of combinatorial test suites. One benefit of
such an abstraction is that they can be studied mathematically and algorithmically.

1.3.1 Latin Squares

In the early 1980s, Robert Mandl [ 30 ] used orthogonal Latin squares in compiler
testing. In particular, the method was used in designing the Ada Compiler Validation
Capability test suite. Later, Williams and Probert [ 43 ] also used orthogonal Latin
squares in the testing of network interfaces.