developer who waits until the evening of the feature freeze can check in code that
causes a 20% regression and deal with it later.
2. Performance characteristics of code will change as the code changes. This is the same
principle that argued for testing the full application (in addition to any module-level
tests that may occur): heap usage will change, code compilation will change, and so
Despite these challenges, frequent performance testing during development is important,
even if the issues cannot be immediately addressed. A developer who introduces code caus-
ing a 5% regression may have a plan to address that regression as development proceeds:
maybe her code depends on some as-yet-to-be integrated feature, and when that feature is
available, a small tweak will allow the regression to go away. That's a reasonable position,
even though it means that performance tests will have to live with that 5% regression for a
few weeks (and the unfortunate but unavoidable issue that said regression is masking other
On the other hand, if the new code causes a regression that can only be fixed with some ar-
chitectural changes, it is better to catch the regression and address it early, before the rest of
the code starts to depend on the new implementation. It's a balancing act, requiring analytic
and often political skills.
Early, frequent testing is most useful if the following guidelines are followed:
All performance testing should be scripted (or programmed, though scripting is usually
easier). Scripts must be able to install the new code, configure it into the full environment
(creating database connections, setting up user accounts, and so on), and run the set of
tests. But it doesn't stop there: the scripts must be able to run the test multiple times, per-
form t-test analysis on the results, and produce a report showing the confidence level that
the results are the same, and the measured difference if they are not the same.
The automation must make sure that the machine is in a known state before tests are run:
it must check that no unexpected processes are running, that the OS configuration is cor-
rect, and so on. A performance test is only repeatable if the environment is the same from
run to run; the automation must take care of that.