Databases Reference
In-Depth Information
• Test your application's ability to tolerate a changing environment. For example,
you can find out how your application performs during a sporadic peak in con-
currency or with a different configuration of servers, or you can see how it handles
a different data distribution.
• Test different hardware, software, and operating system configurations. Is RAID
5 or RAID 10 better for your system? How does random write performance change
when you switch from ATA disks to SAN storage? Does the 2.4 Linux kernel scale
better than the 2.6 series? Does a MySQL upgrade help performance? What about
using a different storage engine for your data? You can answer these questions with
special benchmarks.
• Prove that your newly purchased hardware is correctly configured. We can't count
the number of times we've used benchmarks to burn in a new system and found
misconfigurations or faulty hardware components. It's a good idea not to put a
new server into production without benchmarking it first, and never to take a
hosting provider or hardware vendor's word for what is installed and how fast it
should perform. Testing is always a good idea, if possible.
You can also use benchmarks for other purposes, such as to create a unit test suite for
your application, but we focus only on performance-related aspects here.
The problem with benchmarking is that it isn't real. The workload you use to stress
the system is usually very simple in comparison with real-life workloads. There's a
reason for that: real-life workloads are nondeterministic, varying, and too complex to
understand readily. If you benchmarked your systems with real workloads, it would be
harder to draw accurate conclusions from the benchmarks.
In what ways is a benchmark's workload unrealistic? There are many artificial dimen-
sions to a benchmark—the data size, the distribution of data and queries—but perhaps
the most important is that a benchmark usually runs as fast as it possibly can, loading
the system so heavily that it behaves badly. In many cases we would like to tell bench-
mark tools to run as fast as possible within certain tolerances, throttling themselves as
necessary to maintain good performance. This would be especially helpful for deter-
mining the system's maximum usable capacity. However, most benchmarking tools
don't support such complexity. It's good to keep in mind that the tools limit the mean-
ingfulness and usefulness of the results.
It's tricky to use benchmarks for capacity planning, too. It is often unrealistic to ex-
trapolate from benchmark results. For example, suppose you want to know how much
business growth you will be able to support with your new database server. You bench-
mark the existing server, then benchmark the new server and find that it can perform
40 times as many transactions per second. But that doesn't mean that your business
will be able to grow 40-fold on the new server. By the time your revenue grows that
much, the system will probably have more traffic, more users, more data, and more
interconnections between related pieces of data. You should not expect any of those
factors to grow only 40 times, especially the number of relationships. In addition, your
