Java Reference
In-Depth Information
Figure 2-3. Set of response times with an outlier
This data set includes a huge outlier: one request took 100 seconds. As a result, the positions
of the 90th% and average response times are reversed. The average response time is a whop-
ping 5.95 seconds, but the 90th% response time is 1.0 second. Focus in this case should be
given to reducing the effect of the outlier (which will drive down the average response time).
Outliers like that are rare in general, though they can more easily occur in Java applications
because of the pause times introduced by GC. (Not that garbage collection should be expec-
ted to introduce a 100-second delay, but particularly for tests with small average response
times, the GC pauses can introduce significant outliers.) In performance testing, the usual fo-
cus is on the 90th% response time (or sometimes the 95th% or 99th% response time; there is
nothing magical about 90%). If you can only focus on one number, a percentile-based num-
ber is the better choice, since achieving a smaller number there will benefit a majority of
users. But it is even better to look at both the average response time and at least one
percentile-based response time, so that you do not miss the case where there are large outli-
Search WWH ::

Custom Search