Databases Reference
In-Depth Information
a few cases where benchmarks with odd results turned out to be completely meaning-
less due to a silly mistake, and the benchmarker looked rather foolish in the end.
7
Running the Benchmark and Analyzing Results
Once you've prepared everything, you're ready to run the benchmark and begin gath-
ering and analyzing data.
It's a good idea to automate the benchmark runs. Doing so will improve your results
and their accuracy, because it will prevent you from forgetting steps or accidentally
doing things differently on different runs. It will also help you document how to run
the benchmark.
Any automation method will do; for example, a Makefile or a set of custom scripts.
Choose whatever scripting language makes sense for you: shell, PHP, Perl, etc. Try to
automate as much of the process as you can, including loading the data, warming up
the system, running the benchmark, and recording the results.
When you have it set up correctly, benchmarking can be a one-step
process. If you're just running a one-off benchmark to check something
quickly, you might not want to automate it, but if you think you'll ever
refer to the results in the future, do it anyway. If you don't, you'll never
remember how you ran the benchmark or what parameters you used,
and you won't be able to use the benchmark results later.
You'll usually run a benchmark several times. Exactly how many runs you need
depends on how you score the results, and how important the benchmark is. If you
need greater certainty, you need to run the benchmark more times. Common practices
are to look for the best result, average all the results, or just run the benchmark five
times and average the three best results. You can be as precise as you want. You might
want to apply statistical methods to your results, find the confidence interval, and so
on, but you often don't need that level of certainty.
8
If it answers your question to your
satisfaction, you can simply run the benchmark several times and see how much the
results vary. If they vary widely, either run the benchmark more times or run it for
longer, which usually reduces variance.
Once you have your results, you need to analyze them—that is, turn the numbers into
knowledge. The goal is to answer the question that frames the benchmark. Ideally,
you'd like to be able to make a statement such as “Upgrading to four CPUs increases
throughput by 50% with the same response time” or “The indexes made the queries
faster.” If you want to be more scientific, read up on the
null hypothesis
before
7. This has never, ever happened to any of the authors. Just in case you're wondering.
8. If you really need scientific, rigorous results, you should read a good book on how to design and execute
controlled tests, because the subject is much larger than we can cover here.
