Databases Reference
In-Depth Information
Unfortunately, setting up a realistic benchmark can be complicated and time-
consuming, and if you can get a copy of the production dataset, count yourself lucky.
It might even be impossible—for example, you might be developing a new application
that has few users and little data. If you want to know how it'll perform when it grows
very large, you'll have no option but to simulate the larger application's data and
workload.
What to Measure
It's best to identify your goals before you start benchmarking—indeed, before you even
design your benchmarks. Your goals will determine the tools and techniques you'll
use to get accurate, meaningful results. Try to frame your goals as a questions, such as
“Is this CPU better than that one?” or “Do the new indexes work better than the current
ones?”
You sometimes need different approaches to measure different things. For example,
latency and throughput might require different benchmarks.
Consider some of the following measurements and how they fit your goals:
Throughput
Throughput is defined as the number of transactions per unit of time. This is one
of the all-time classics for benchmarking database applications. Standardized
benchmarks such as TPC-C (see
http://www.tpc.org
)
are widely quoted, and many
database vendors work very hard to do well on them. These benchmarks measure
online transaction processing (OLTP) throughput and are most suitable for inter-
active multiuser applications. The usual unit of measurement is transactions per
second, although it is sometimes transactions per minute.
Response time or latency
This measures the total time a task requires. Depending on your application, you
might need to measure time in micro- or milliseconds, seconds, or minutes. From
this you can derive aggregate response times, such as average, maximum, mini-
mum, and percentiles. Maximum response time is rarely a useful metric, because
the longer the benchmark runs, the longer the maximum response time is likely to
be. It's also not at all repeatable, because it's likely to vary widely between runs.
For this reason, it's common to use
percentile response times
instead. For example,
if the 95th percentile response time is 5 milliseconds, you know that the task fin-
ishes in 5 milliseconds or less 95% of the time.
It's usually helpful to graph the results of these benchmarks, either as lines (for
example, the average and 95th percentile) or as a scatter plot so you can see how
the results are distributed. These graphs help show how the benchmarks will be-
have in the long run. We will return to this point later in this chapter.
