Game Development Reference
In-Depth Information
From this information, tests 18, 30, and 34 could be run next to help identify the pair
that causes the defect. If none of those tests fail, then the cause is narrowed down to
the first three pairs, which are only found in row 13: 50-1, 50-Large, or 50-Easy. If test
18 fails, then look for the 1-Easy pair to be the cause of the problem. Likewise, if test
30 fails then suspect the 1-Large combination. If test 34 fails you can suggest Large-
Easy as the cause of the problem in your defect report.
The Allpairs output file is tab delimited so you can paste it right into Microsoft Excel or
any other program supporting that format. The Allpairs tool files and the examples from
this chapter, including the complete output file, are provided on the topic's CD-ROM.
The examples used in this chapter have produced tables with significant efficiency,
covering hundreds of potential combinations in no more than a few dozen tests. As it
turns out, these are very modest examples. Some configurations can yield reductions
of more than 100:1, 1000:1, and even beyond 1,000,000:1. It all depends on how many
parameters you use and how many test values you specify for each parameter. But do
you always want to do
less
testing?
Some game features are so important that they deserve more thorough testing than
others. One way to use pairwise combinatorial tests for your game is to do full com-
binatorial testing for critical features, and pairwise for the rest. Suppose you identify
10% of your game features as “critical�? and that each of these features has an average
of 100 tests associated with them (approximately a 4x4x3x2 matrix). It is reasonable
to expect that the remaining 90% of the features could be tested using pairwise com-
binatorial tables, and only cost 20 tests per feature. The cost of full combinatorial testing
of all features is 100*N, where N is the total number of features to be tested. The cost
of pairwise combinatorial testing 90% of those features is 100*0.1*N + 20*0.9*N =
10*N+18*N = 28*N. This provides a 72% savings by using pairwise for the non-critical
90%.
Another way to use combinatorial tests in your overall strategy is to create some tables
to use as “sanity�? tests. The number of tests you run early in the project will stay low,
and then you can rely on other ways of doing “traditional�? or “full�? testing once the
game can pass the sanity tests. Knowing which combinations work properly can also
help you select which scenarios to feature in pre-release videos, walkthroughs, or pub-
lic demos.
