large number of loops, but the per-iteration difference is often measured in nanoseconds. Yes,
nanoseconds add up, and “death by 1,000 cuts” is a frequent performance issue. But particu-
larly in regression testing, consider whether tracking something at the nanosecond level actu-
ally makes sense. It may be important to save a few nanoseconds on each access to a collec-
that occurs less frequently—for example, maybe once per request for a servlet—fixing a
nanosecond regression found by a microbenchmark will take away time that could be more
profitably spent on optimizing other operations.
Writing a microbenchmark is hard. There are very limited times when it can be useful. Be
aware of the pitfalls involved, and make the determination if the work involved in getting a
reasonable microbenchmark is worthwhile for the benefit—or if it would be better to concen-
trate on more macro-level tests.
The best thing to use to measure performance of an application is the application itself, in
conjunction with any external resources it uses. If the application normally checks the cre-
dentials of a user by making LDAP calls, it should be tested in that mode. Stubbing out the
LDAP calls may make sense for module-level testing, but the application must be tested in
its full configuration.
As applications grow, this maxim becomes both more important to fulfill and more difficult
to achieve. Complex systems are more than the sum of their parts; they will behave quite dif-
ferently when those parts are assembled. Mocking out database calls, for example, may mean
that you no longer have to worry about the database performance—and hey, you're a Java
person; why should you have to deal with someone else's performance problem? But data-
base connections consume lots of heap space for their buffers; networks become saturated
when more data is sent over them; code is optimized differently when it calls a simpler set of
methods (as opposed to the complex code in a JDBC driver); CPUs pipeline and cache short-
er code paths more efficiently than longer code paths; and so on.
The other reason to test the full application is one of resource allocation. In a perfect world,
there would be enough time to optimize every line of code in the application. In the real
world, deadlines loom, and optimizing only one part of a complex environment may not
yield immediate benefits.
Consider the data flow shown in Figure 2-1 . Data comes in from a user, some proprietary
business calculation is made, some data based on that is loaded from the database, more pro-
prietary calculations are made, changed data is stored back to the database, and an answer is