Databases Reference
In-Depth Information
application will almost certainly have changed by the time your revenue has grown by
a factor of 40. You will have new features, some of which might impact the database
far out of proportion to their apparent complexity. These changes in workload, data,
relationships, and features are very hard to simulate, and their impacts are hard to guess.
As a result, we usually settle for approximations, with a goal of knowing whether there's
still a decent amount of spare capacity in the system. It is possible to do more realistic
load testing (as distinct from benchmarking), but it requires a lot of care in creating the
dataset and workload, and in the end it's not really a benchmark. Benchmarks are
simpler, more directly comparable to each other, and cheaper and easier to run. And
despite their limitations, benchmarks are useful. You just need to be clear about what
you're doing and in what ways the outcome is meaningful.
Benchmarking Strategies
There are two primary benchmarking strategies: you can benchmark the application
as a whole, or isolate MySQL. We call these two strategies
full-stack
and
single-
component
benchmarking, respectively. There are several reasons to measure the ap-
plication as a whole instead of just MySQL:
• You're testing the entire application, including the web server, the application
code, the network, and the database. This is useful because you don't care about
MySQL's performance in particular; you care about the whole application.
• MySQL is not always the application bottleneck, and a full-stack benchmark can
reveal this.
• Only by testing the full application can you see how each part's cache behaves.
• Benchmarks are good only to the extent that they reflect your actual application's
behavior, which is hard to do when you're testing only part of it.
On the other hand, application benchmarks can be hard to create and even harder to
set up correctly. If you design the benchmark badly, you can end up making bad deci-
sions, because the results don't reflect reality.
Sometimes, however, you don't really want to know about the entire application. You
might just need a MySQL benchmark, at least initially. Such a benchmark is useful if:
• You want to compare different schemas or queries.
• You want to benchmark a specific problem you see in the application.
• You want to avoid a long benchmark in favor of a shorter one that gives you a faster
“cycle time” for making and measuring changes.
It's also useful to benchmark MySQL when you can repeat your application's queries
against a real dataset. The data itself and the dataset's size both need to be realistic. If
possible, use a snapshot of actual production data.
