Tools and Benchmarks - Automatic Generation of Combinatorial Test Data - page 77

•

CASA ( http://cse.unl.edu/~citportal/ ) is a tool for generating Covering Arrays by
Simulated Annealing [ 7 ].

• Cascade ( http://lcs.ios.ac.cn/~zj/ct.html ) is based on the one-test-at-a-time strat-

egy, and it uses pseudo-Boolean optimization (PBO) techniques. See Sect. 3.4 .
The tool can search for variable-strength covering arrays with constraints.

• CitLab ( http://code.google.com/a/eclipselabs.org/p/citlab/ ) [ 2 ] is a Laboratory for

Combinatorial Interaction Testing [ 6 ]. It provides an abstract language for com-
binatorial testing problems, so as to promote the interoperability among test data
generation tools, and to make the exchange of models and data more convenient.

• EXACT [ 14 ] ( http://lcs.ios.ac.cn/~zj/ct.html ) tries to find a small covering array

by backtracking search. For more details, see Chap. 6 . EXACT has been used to
find several arrays, which are smaller than what people knew.

• Jenny ( http://burtleburtle.net/bob/math/jenny.html ) is an open-source tool for pair-

wise testing, implemented in C.

• PICT (Pairwise independent combinatorial testing) is a widely used test generator,

developed by Jacek Czerwonka (Microsoft Corp.) [ 5 ].

• QICT [ 12 ] is an open-source pairwise test data generator, implemented in C#.

• Testcover.com ( http://testcover.com/ ) provides a test case generator for pairwise

testing. It is also available as a web service.

• Test Vector Generator (TVG) is available at http://sourceforge.net/projects/tvg .

When we need to use combinatorial testing in a project, which tools shall we
choose? How do the above tools compare with each other?

Researchers have some general observations. For instance, greedy techniques
tend to be faster, while meta-heuristic search techniques can often find smaller test
suites. There are also some empirical evaluations of the different approaches, e.g.,
Grindal et al. [ 8 ]. However, as more methods and tools are being developed, and with
the advance in hardware performance, the situation may change. Another factor to
consider is that some algorithms involve a certain kind of randomness. If you execute
the tools several times, you may get several different results.

Thus, it seems to be difficult to give a simple answer saying that one tool is better
than another. We encourage the readers to try the available tools on well-known
benchmarks and on your own applications.

7.2 Applications and Benchmarks

Combinatorial testing is closely related to combinatorics, which is a branch of
mathematics. Mathematicians have publishedmany papers on combinatorial designs.
But there are still many open questions. On the other hand, as a software/system
testing technique, combinatorial testing has been used in practice by quite some
researchers and engineers.