Testing Concurrent Programs - Java Concurrency in Practice - page 308

Figure 12.4. Completion Time Histogram for TimedPutTakeTest with Single-item Buffers.
So, unless threads are continually blocking anyway because of tight synchronization require-
ments, nonfair semaphores provide much better throughput and fair semaphores provides
lower variance. Because the results are so dramatically different, Semaphore forces its cli-
ents to decide which of the two factors to optimize for.
12.3. Avoiding Performance Testing Pitfalls
In theory, developing performance tests is easy—find a typical usage scenario, write a pro-
gram that executes that scenario many times, and time it. In practice, you have to watch out
for a number of coding pitfalls that prevent performance tests from yielding meaningful res-
ults.
12.3.1. Garbage Collection
The timing of garbage collection is unpredictable, so there is always the possibility that the
garbage collector will run during a measured test run. If a test program performs N iterations
and triggers no garbage collection but iteration N + 1would trigger a garbage collection, a
small variation in the size of the run could have a big (but spurious) effect on the measured
time per iteration.