Knowledge Creation - Designing Games - page 318

strategy, but if one of the million players in the public beta does, it will
show up as a spike in the data.
Sometimes we can use cleverly designed metrics to gather data that
seems impossible to get. For example, during the development of Halo:
Reach , Bungie's designers wanted to learn more about the effect Internet
lag has on the experience. The game was already instrumented to record
a movie file including every game variable at all times, including lag. But
this wasn't enough—the designers wanted to know how players perceived
the lag, which isn't easy to see in the data. Playtesting couldn't solve this,
since many lag problems are so rare that it would require massive amounts
of time for designers to watch enough playtests to catch them. Worse, real-
istic net testing requires players to be physically spread out, which makes
traditional playtest protocols nearly impossible to follow.
They solved the problem by adding a special button that reports, “I
just saw lag.” These button presses were recorded along with the rest of
the game. Once the data started coming in, the designers could watch the
movies and skip to the points where players perceived lag to see exactly
what was on their screen. This ingenious method brought them a huge
amount of reliable knowledge about lag perception with relatively little de-
veloper effort. They solved many subtle lag perception issues this way, and
Halo: Reach had great online play. They had great net code, but it wasn't
because they were network-coding whiz kids. Like Valve, they worked with
more knowledge.
inventeD metHoDs
Some questions can't be answered by established methods. In these cases,
we must go back to first principles and invent a new way to get the knowl-
edge we need.
When making a game for the elderly, a designer might have to invent
a special playtest protocol. A game with an unusual in-game economy
might require new ways of analyzing and interpreting data. A design team
spread across continents might have to debate and brainstorm differently
from one that collocates. Inventing and refining our knowledge creation
tools is an integral part of game design.
The Organic Process
Let's look at a classic example of a process of knowledge creation. This
example isn't from game design. It's from invention.