to platform. Caches—software and, more importantly, hardware—behave differently on
different systems, and under different loads. And so on…
Hence, the performance of a particular production environment can never be fully known
without testing the expected load on the expected hardware. Approximations and extra-
polations can be made from running smaller tests on smaller hardware, and in the real
world, duplicating a production environment for testing can be quite difficult or expens-
ive. But extrapolations are simply predictions, and even in the best case, predictions can
be wrong. A large-scale system is more than the sum of its parts, and there can be no sub-
stitute for performing adequate load testing on the target platform.
1. Frequent performance testing is important, but it doesn't occur in a vacuum; there
are trade-offs to consider regarding the normal development cycle.
2. An automated testing system that collects all possible statistics from all machines
and programs will provide the necessary clues to any performance regressions.
Performance testing involves a number of trade-offs. Making good choices among compet-
ing options is crucial to successfully tracking the performance characteristics of a system.
Choosing what to test is the first area where experience with the application and intuition
will be of immeasurable help when setting up performance tests. Microbenchmarks tend to
be the least helpful test; they are useful really only to set a broad guideline for certain opera-
tions. That leaves a quite broad continuum of other tests, from small module-level tests to a
large, multitiered environment. Tests all along that continuum have some merit, and choosing
the tests along that continuum is one place where experience and intuition will come into
play. However, in the end there can be no substitute for testing a full application as it is de-
ployed in production; only then can the full effect of all performance-related issues be under-
Similarly, understanding what is and is not an actual regression in code is not always a black-
and-white issue. Programs always exhibit random behavior, and once randomness is injected
into the equation, we will never be 100% certain about what data means. Applying statistical