New Challenges in Physical Database Design - Automated Physical Database Design and Tuning - page 231

100

80

C1
C2
C3

60

40

20

0

0 1000 2000 3000

Time

(a) M metric for a 22-query TPC-H workload.

2000

1500

1000

500

0

-500

Queries

metric for a 22-query TPC-H workload.
FIGURE 12.5 Metrics to compare physical design tuners. (Used with per-
mission from Bruno, N. SIGMOD Record , 36(4), 2007).

(b) I

than two configurations. are complementary metrics that provide

different insights when comparing physical design tuners.
In the rest of this section we comment on some important aspects that
should be considered while designing an evaluation metric.

M and I

Actual vs. estimated cost. An important question is whether to use the
actual cost of executing queries in the workload or the estimated cost by
the optimizer. In the context of evaluating a full system (i.e., not only
the tuning tool but also the query optimizer, query processor, and even
the underlying operating system), the actual query cost is clearly the
best, most unbiased choice. However, if the purpose of the benchmark is
an isolated evaluation of different physical design tools, execution costs