Java Reference
In-Depth Information
QUICK SUMMARY
1. Correctly determining whether results from two tests are different requires a level
of statistical analysis to make sure that perceived differences are not the result of
random chance.
2. The rigorous way to accomplish that is to use Student's t-test to compare the res-
ults.
3. Data from the t-test tells us the probability that a regression exists, but it doesn't
tell us which regressions should be ignored and which must be pursued. Finding
that balance is part of the art of performance engineering.
Test Early, Test Often
Fourth and finally, performance geeks (including me) like to recommend that performance
testing be an integral part of the development cycle. In an ideal world, performance tests
would be run as part of the process when code is checked into the central repository; code
that introduces performance regressions would be blocked from being checked in.
There is some inherent tension between that recommendation and other recommendations in
this chapter—and between that recommendation and the real world. A good performance test
will encompass a lot of code—at least a medium-sized mesobenchmark. It will need to be re-
peated multiple times to establish confidence that any difference found between the old code
and the new code is a real difference and not just random variation. On a large project, this
can take a few days or a week, making it unrealistic to run performance tests on code before
checking it into a repository.
The typical development cycle does not make things any easier. A project schedule often es-
tablishes a feature-freeze date: all feature changes to code must be checked into the reposit-
ory at some early point in the release cycle, and the remainder of the cycle is devoted to
shaking out any bugs (including performance issues) in the new release. This causes two
problems for early testing:
1. Developers are under time constraints to get code checked in to meet the schedule;
they will balk at having to spend time fixing a performance issue when the schedule
has time for that after all the initial code is checked in. The developer who checks in
code causing a 1% regression early in the cycle will face pressure to fix that issue; the
Search WWH ::




Custom Search