New Challenges in Physical Database Design - Automated Physical Database Design and Tuning - page 229

an open area of research. Useful benchmarks should contain databases and
workloads taken from at least the following three sources:
Micro-benchmarks: These are very specific databases and workloads that
evaluate the different DBMS capabilities and for which optimal config-
urations can be manually derived.
Synthetic benchmarks: These involve complex workloads that exercise the
full capabilities of the underlying query processor and cannot be man-
ually analyzed. An example of a synthetic benchmark in the broader
context of query processing is the TPC-H benchmark.
Real benchmarks: These involve databases and workloads taken (or
adapted) from real-world scenarios, which can address subtle issues over-
looked by the previous two approaches.
12.4.2 Baseline Configuration
Baseline configurations can be useful for standardizing results. Different al-
ternatives are compared against a baseline, which in turn allows us to easily
compare different approaches. We next discuss some ways to define baseline
configurations.
Base configuration: The base configuration is the one that contains no in-
dexes except for those required by constraints (e.g., uniqueness or foreign
keys). This configuration is the worst possible one for select statements
(since no indexes are available other than those that must always exist)
and the best possible one for update shells (since the minimum number
of indexes needs to be updated). The base configuration is also the one
that has the smallest space overhead and can always be generated, as it
is independent of the workload instance.
Standard indexes: A slightly more complex baseline involves considering
index templates. For instance, we can consider configurations that have
all possible single-column indexes available. These configurations still
have the advantage that they are independent of the workload and
therefore can be programmatically generated for arbitrary benchmark
instances. Except for heavy-update workloads, these configurations gen-
erally result in a more reasonable baseline than the base configuration.
Manually tuned configuration: An interesting baseline results from man-
ually tuning the database for the given workload by an expert DBA. In
contrast to previous alternatives, this baseline depends on the specific
database and workload and therefore cannot be generated programmat-
ically. It is, however, a more realistic alternative to compare different
approaches.