Databases Reference
In-Depth Information
Figure 2-1. I/O performance during an extended benchmark
As the system warmed up, the read I/O activity settled into a steady line after three or
four hours, but writes remained variable for at least eight hours, and then there were a
few sharp notches in the plot of writes. After that, both reads and writes seemed to
settle in.
4
A rule of thumb is to wait until the system looks like it's been steady for at
least as long as the initial warmup appeared to take. We ended up running this bench-
mark for 72 hours to ensure that the system was exhibiting its typical long-term
behavior.
A very common benchmarking mistake is to run a series of short benchmarks, such as
60-second runs, and conclude something about the system's performance from that.
We hear a lot of comments such as “I tried benchmarking the new version of the server,
and it wasn't faster than the old version.” When we dig into the actual benchmark, we
often find the benchmarks were conducted in a way that doesn't support the conclu-
sions they're intended to generate. Sometimes people protest that they just don't have
time to benchmark the server for 8 or 12 hours at 10 different levels of concurrency on
two or three server versions. If you don't have the time to do the benchmarks right, any
time you do spend is wasted; it is better to trust other people's results, instead of doing
an incomplete benchmark and getting the wrong answers.
4. By the way, the graph of write I/O activity shows extremely bad behavior; this system's steady state is a
performance catastrophe. Calling it a “steady state” is almost laughable, but our point is that it's indicative
of how the server is going to behave over the long term.
