Assessments - The Practice of Cloud System Administration - page 444

• Both a red flag level of toil and a target level are defined. The red flag is lowered
when toil reaches the target level.
• During the red flag period, the team works on only the highest-impact toil-reduc-
tion projects.
• During the red flag period, the team has management support for putting other pro-
jects on hold until toil is reduced to a target level.
• After each step in a project, statistics on toil are closely monitored, providing feed-
back on any positive or negative changes.
Level 4: Managed
• Project time versus toil is tracked on a dashboard, and the amount of time spent on
each individual project or manual task is also tracked.
• Red flags are raised automatically, and the dashboard gives an overview of where
the problems lie.
• The time-tracking data is monitored for trends that give an early alert for teams
that are showing an increase in toil in one or more areas.
• KPIs are defined and tracked to keep toil within the desired range and minimize
the red flag periods.
Level 5: Optimizing
• The target and red flag levels are adjusted and the results are monitored to the ef-
fect on overall flow, performance, and innovation.
• Changes to the main project prioritization process are introduced and evaluated for
positive or negative impact, including the impact on toil.
• Changes to the red flag toil-reduction task prioritization process are introduced and
evaluated.
A.12 Disaster Preparedness
An operations organization needs to be able to handle outages well, and it must have prac-
tices that reduce the chance of repeating past mistakes. Disasters and major outages hap-
pen. Everyone in the company from the top down needs to recognize that fact, and adopt
a mind-set that accepts outages and learns from them. Systems should be designed to be
resilient to failure.