Creating KPIs - The Practice of Cloud System Administration - page 397

19.3.3 Evaluating the KPI
Afewweeksafterdeployment,theinitialresultsshouldbeauditedforunintendednegative
side effects.
In our example, depending on how often VMs are created, the resulting KPI measure-
ments might be reviewed daily, weekly, or monthly. These results might inspire additional
metrics to isolate problem areas. The VM creation process is made up of many steps. By
measuringthewaittimebeforeeachstep,aswellasthedurationofeachstep,opportunities
for improvement can be easily found.
For example, measuring lead time might reveal a long span of time between when a
ticketisfiledandwhentheactualcreationbegins,indicatingthataself-servicerequestsys-
tem would have a large benefit. Lead time might reveal that requests are often delayed for
weeks waiting for new resources to be installed, indicating that capacity planning needs
improvement. Collecting data on how many times a task fails and is retried might indicate
thattheOSinstallationprocessisunstableandshouldbeimproved.PerhapsDNSpropaga-
tion delays can be tightened or the OS installation process can be made faster using image-
based installations. Monitoring network utilization might find an overloaded link that, if
improved, could result in faster installation time in general. By analyzing the types of re-
quests, it may be determined that a few standard-size VMs can be pre-created and simply
handed out as needed.
All these changes are viable. By taking measurements, we can predict how each one
might improve the KPI. By collecting KPI measurements before and after changes, we can
measure the actual improvement.
19.4 Case Study: Error Budget
Benjamin Treynor Sloss, Vice President of Engineering at Google, revealed a highly suc-
cessful KPI called Google Error Budget. The goal was to encourage high uptime without
stifling innovation, and to encourage innovation without encouraging undue risk.
19.4.1 Conflicting Goals
There is a historic conflict between developers and operations teams. Developers want to
launch new features; operations teams want stability.
Developers are in the business of making change. They are rewarded for new features,
especiallyonesthatarehighlyvisibletotheendcustomers.Theywouldprefertohaveeach
feature they create pushed into production as fast as possible so as not to delay gratifica-
tion. The question they get the most from management is likely to be, “When will it ship?”