Creating KPIs - The Practice of Cloud System Administration - page 396

come on line. That said, if capacity planning becomes an issue, we can later establish KPIs
specifically related to capacity planning efficiency.
In Step 3, we imagine how behavior will change. If a cancelled request does not count
towardtheKPI,wecouldsimplycancel anyrequestthattakestoolong.Forexample, ifre-
questsaremadeviaahelpdeskticket,iftheuserdoesn'tsupplyalltheinformationrequired
to complete the task, we could cancel the request and let the user know what information
we need when the user creates the new ticket. This would be terrible behavior but would
improve the KPI.
If the OS installation step is unreliable and requires multiple restarts, this would delay
the total creation time. To improve our KPI numbers, the operators could simply cancel
jobs that fail rather than retry them. Again, this would benefit the KPI but not the user.
We could work around this loophole by adding significant complexity to the KPI.
However, often it is easier to prevent bad behavior by letting people know they are being
watched.First,wecanpublishthenumberofcustomer-initiated cancellation requests.This
permitsmanagementtoquietlywatchforshenanigans.Second,wecanprivatelyagreewith
the team's manager that such behavior will be discouraged. Executive management can as-
sist by creating an environment with high standards, including only hiring managers who
wouldn't tolerate such behavior. Such managers would, for example, notice that the sys-
tem doesn't log who cancelled a request, and require that this be changed so that the KPI
canproperlymeasurecustomer-initiatedcancellationsversuscancellationsfromoperation-
al staff.
Step 4, Revise and Select, results in the following KPI:
The 90th percentile creation time, measured from when the request is received from
the user until the user is able to log into the machine. Manual and automatic retries
after a failure are counted as part of the original request, not as separate requests.
Requests that are outright canceled will be logged and investigated if within a week
there are more than 5 operator-initiated cancellations or if more than 1 percent of all
requests end in customer-initiated cancellation.
Step 5 deploys the KPI. The KPI is communicated to key stakeholders. Measurements
requiredtocalculatetheKPIplusthecomputationitselfshouldbeautomatedandpresented
inadashboard.Inourexample,thesystemmightgeneratethemetricwerequireoritmight
timestamp the request and completion times, requiring some post-processing to correlate
the two and calculate the duration. However the metric is gathered, it should then be stored
andmadeavailable towhateverdatavisualization systemisinusesothatagraphicaldash-
board can be created.