Automation - The Practice of Cloud System Administration - page 247

12.1.5 Lessons Learned
Efforts to automate can backfire. Automation is generally viewed as a pure engineering
task,sothehumancomponentmayalltoooftenbeneglected.Systemsthataimtoeliminate
boring and tedious tasks so that people can tackle more difficult tasks leave the hardest
parts to humans because they are too complex to be automated. Thus mental fatigue due
to many tedious tasks is eliminated, but it is replaced by an even more burdensome mental
fatigue due to the need to tackle difficult problems on a continual basis.
Automation can bring stability to a system, yet this stability results in operators be-
coming less skilled in maintaining the system. Emergency response becomes particularly
brittle, a subject that we will address in Chapter 15 .
When designing automation, askyourself which view ofthe human component isbeing
assumedbythisautomation.Arepeopleabottleneck,asourceofunwantedvariability,ora
resource?Ifpeopleareabottleneck,canyouremovethebottleneckwithoutremovingtheir
visibility into what the system is doing, and without making it impossible for them to ad-
justhowthesystemworks,whennecessary?Ifpeopleareasourceofunwantedvariability,
then you are constraining the environment and inputs of the automation to make it more
reliable. What effect does that have on the people running the automation? How does it
constrain their work? How can exceptions be handled? If people are to be a resource, then
youneedclosercouplingbetweenthepeopleandtheautomation.Butinthatcase,thereare
twothornyissues:whodoeswhichtasks(allocation) andwhoisincharge(responsibility)?
Thelong-termoperationofasystemcanbebrokendownintofourstages:tracking,reg-
ulating, monitoring, and targeting. Tracking covers event detection and short-term control
in response to inputs or detected events. Automation typically starts at this level. Regula-
tion covers long-term control, such as managing transition between states. The case study
in Section 12.3 is an example of automation at the regulation level. Monitoring covers
longer-term controls, interpreting system state and selecting plans to remain within the de-
sired parameters. For example, capacity planning, hardware evaluation and selection, and
similar tasks would fall into this category. Targeting covers setting the overall goals for the
system based on the overall corporate goals—for example, using key performance indic-
ators (KPIs; see Chapter 19 ) to drive the desired behavior. As you move up the chain, the
higher-level tasks are generally more suited to people than to machines. As you automate
further up the chain, people become more disconnected from what the system is doing and
why, and the long-term health of the system as a whole may be jeopardized.