Testing Concurrent Programs - Java Concurrency in Practice - page 309

There are two strategies for preventing garbage collection from biasing your results. One is
to ensure that garbage collection does not run at all during your test (you can invoke the JVM
with -verbose:gc to find out); alternatively, you can make sure that the garbage collector
runs a number of times during your run so that the test program adequately reflects the cost
of ongoing allocation and garbage collection. The latter strategy is often better—it requires a
longer test and is more likely to reflect real-world performance.
Most producer-consumer applications involve a fair amount of allocation and garbage collec-
tion—producers allocate new objects that are used and discarded by consumers. Running the
bounded buffer test for long enough to incur multiple garbage collections yields more accur-
ate results.
12.3.2. Dynamic Compilation
Writing and interpreting performance benchmarks for dynamically compiled languages like
Java is far more difficult than for statically compiled languages like C or C++. The HotSpot
JVM (and other modern JVMs) uses a combination of bytecode interpretation and dynamic
compilation. When a class is first loaded, the JVM executes it by interpreting the bytecode.
At some point, if a method is run often enough, the dynamic compiler kicks in and converts
it to machine code; when compilation completes, it switches from interpretation to direct ex-
ecution.
The timing of compilation is unpredictable. Your timing tests should run only after all code
has been compiled; there is no value in measuring the speed of the interpreted code since
most programs run long enough that all frequently executed code paths are compiled. Allow-
ing the compiler to run during a measured test run can bias test results in two ways: com-
pilation consumes CPU resources, and measuring the run time of a combination of inter-
preted and compiled code is not a meaningful performance metric. Figure 12.5 shows how
this can bias your results. The three timelines represent the execution of the same number of
iterations: timeline A represents all interpreted execution, B represents compilation halfway
through the run, and C represents compilation early in the run. The point at which compila-
tion runs seriously affects the measured per-operation runtime. [7]