Driving to the Finish - Game Coding Complete - page 853

Smoking the Build at Red Fly Studio

Red Fly Studio never had enough of anything we needed, especially testers.
Our typical game took about five hours to play all the way through, which
divided among three or four testers meant that each new build of our game
took more than an hour for our testers to do a quick playthrough. Combine
that with the problem of testing on multiple platforms or in multiple
languages, and the testing time went up pretty fast. Because a build-
breaking bug could happen at any time, we decided to have the entire team
join in and help the testers play through the entire game, in every language.
Split this job into as many as 20 or 30 developers and even our longer games
got smoke tested in about 20 minutes.

A Full-Time Job

At EA, we have a complicated build promotion process. Whenever you check
something in, the build machine will sync up and build it. If the build passes,
your change gets promotion to

“ latest, ” which means if anyone syncs to the

“ data through the data tool, they will get your changes. Everyday, a

few QA testers are assigned the task of promoting the last

latest ”

“ latest ” data to

“ A complete smoke test is run with an

established test plan. The whole process takes several hours. Once the
testers sign off, the build is promoted. Anyone can grab the

LKG, ” or “ Last Known Good. ”

“ LKG ” data

and have a decent, working copy of the game.

Later in the project, this turns into a full-time job. We had one or two testers
on The Sims Medieval during the last six months or so who would do
nothing but LKG testing. It took both of them the entire day to run through the whole game and write
up bugs. When attempting to promote a major milestone like alpha, the entire test team was dedicated to
running through the entire game, including each play path for the dozens of quests.

There are a lot of good ways to measure how important a particular bug is, one of
which is user pain. Look for this blog article written in 2008 on it: http://www.lost-
garden.com/search/label/User%20Pain. Basically, it measures a bug on many dimen-
sions, such as what kind of bug it is, whether it blocks progress in the game, and how
often it happens. This is boiled down to a number, the calculation of which is
completely up to the team and what they feel is important.
I use a slightly different approach and measure bugs in four categories:

n Class AA: This is “ drop everything you are doing and fix this bug, ”
as it is significantly hampering the team from getting testing or work
done.
n Class A: This bug must be fixed or the game can

' t ship. It might be a persistent

crash during a level load, for example.