Game Development Reference
Adding automatic testing tools creates a heightened expectation in upper management.
Suddenly, the fact that automatic testing tools are being used can lead managers to
have unrealistic expectations about lower production costs, reduced development
times, and so on. Indeed, this exact point is a key hurdle facing any test automation
advocate in a company. How do you introduce something to management that in the
near term might significantly add to cost and production time, but that holds out the
promise of potential gains down the road in titles that may not see the light of day for
some seasons to come? If there is sufficient stability and commonality in your typical
game code or the types of games you produce, then perhaps you will have a clear argu-
ment. If your automated test code, scripts, and tools need to be extensively rewritten for
each new game, your argument about the long-term payoff will be greatly diminished.
It is vital that you set management expectations accurately and reasonably when proposing
to automate even a small part of your testing process. In doing so, you may find you are up
against ingrained beliefs about automated testing, the result of media hype, misconceptions
of many kinds, and more. Here are some of the false expectations you may face:
Everything will be automated. For the vast majority of companies it just will
not make sense to automate everything.
Automation will lead to lower costs and greater efficiency. While greater
efficiency is hopefully gained where automation is used properly, the fact is
that, in the near term at least, costs will rise as extra lines of code and scripts
need to be written, management of code increases, greater volume of data is
generated that has to be managed and considered, and so on.
All manual tests will be automated. Much of what is being done manually will
not be suitable for automation.
Tools will be purchased off the shelf. In reality, if your company uses a lot of
proprietary game code, your test code and test scripts will probably be custom
No ramp-up time will be needed. For some reason, automation seems to carry
with it the connotation for some of no ramp-up, but the opposite is the case.
Financial benefits will be immediate. There will be an expectation of instant
payback, when in fact it may be several titles down the road before the full
positive effects of automation are realized.
Automated defect reporting will remove the need for human staff. Without
human involvement, teams soon discover that automated systems can generate
a staggering amount of data. Much of that data should not be sent to the team,
because it consists of false positives, false negatives, or merely duplicate data.