Game Development Reference
In-Depth Information
Regular Regression
Because of the nature of distributed systems, new code will regularly break old code.
For this reason, the TSO automated test system had to be able to validate every game
feature to ensure stability of the release game. Its regression engine was linked closely
to the outcome of normal QA procedures, ensuring that wherever a new defect was
observed, the regression test client would focus on suspected trouble spots and generate
useful data to completely eradicate the bug. Thus, critical path roadblocks were quickly
identified and removed. Interestingly, the team noted a large number of false positives
and false negatives generated by the system, so the importance of human viewing of
test result data was emphasized to ensure there would be no wasted time chasing items
that could be overlooked.
New Use for Old Code
Surprisingly, the team reported that relatively little new C++ code was required to
support their testing system. This was largely achieved by utilizing the existing game
code—the game client code in particular—and reformatting it as test code. Similarly,
reusing the actual game GUI and reconfiguring it for test purposes kept new coding
to a minimum. Indeed, the basis for the test code was found in existing code already in
the game that had been put there to enable normal testers and programmers to “cheat.�?
All for One and One for All
Many of the TSO test scripts are relatively short. Scripting all required test conditions
by hand was not realistic, so an extensible series of algorithms was used to generate
events wherever possible. Random and deterministic strategies were used to generate
scripts on-the-fly to replace the otherwise lengthy hand coding that would have been
required. For instance, the TSO team introduced an algorithm they called
TestAll
that
would walk through the list of all objects currently in a location, build a list of all pos-
sible interactions, and then execute all possible actions using a deterministic strategy.
They were then able to generalize the results by having the system place the objects in
various locations throughout the game world and retest the objects in a host of new
terrain or location configurations. Using this approach, the team had 230 game
objects under regression and about 40% of the game's supporting infrastructure
under automated regression within a month of starting.
