Game Development Reference
In-Depth Information
thinks the game works even though it doesn't. He hasn't just failed to gain
knowledge—he's gained knowledge that isn't true.
I once interviewed a senior designer on a failed multiplayer shooter.
This was his test protocol: a group of players sitting in a room with food,
playing the game for extended lengths of time. In this environment, the
game seemed to be working well. They iterated, found problems, tested,
and polished the game until it was as deep and balanced as a philosopher
on a tightrope. But that success was deceptive, because their test protocol
did not find any of the design failures that occur when the game is played
by strangers over the Internet instead of friends in the same room. Played
between well-coordinated, highly communicative teams, the game shone.
But online, it collapsed. It was so dependent on intricate team tactics that it
did not function when played by lazy, incompetent strangers. The designers
playtested, but their faulty test protocol hid critical flaws in the design, and
so the game failed in the market and with most of its players.
There are countless ways for test protocols to fail.
Unblinded tests
introduce expectation biases in the playtesters. Group tests create social
competition and copycat opinions among players. Telling players to think
aloud helps designers interpret players' actions, but also changes those
players' actions. Tester selection introduces biases that will hide problems
that only appear when people of specific ages, genders, cultures, or skill
levels play the game. Small numbers of testers mean our data is skewed by
surprisingly large random statistical variances.
In the end, we can never totally avoid these faults. Test protocol isn't a
matter of right and wrong. It's a craft in which a designer tries to get the
most useful knowledge possible with a given set of resources.
Let's look at some basic test protocols.
self-testing
The cheapest test protocol is to play alone. Even though the designer's play
is biased by his knowledge of the game, just watching the game systems
in motion brings a tremendous amount of understanding. It reveals many
problems in flow, pacing, and balance. And, of course, technical bugs are
best found in self-tests. The earliest loops of an iterative process should
conclude with self-tests.
