Operations in a Distributed World - The Practice of Cloud System Administration - page 160

made to assure that emergency issues receive immediate attention and non-project custom-
er requests are triaged and worked in a timely manner.
More specifically, at any given moment, the highest priority for one person on the team
should be responding to emergencies, the highest priority for one other person on the team
should be responding to normal requests, and the rest of the team should be focused on
project work.
This is counter to the way operations teams often work: everyone running from emer-
gency to emergency with no time for project work. If there is no effort dedicated to im-
provingthesituation,theteamwillsimplyrunfromemergencytoemergencyuntiltheyare
burned out.
Major improvements come from project work. Project work requires concentration and
focus. If you are constantly being interrupted with emergency issues and requests, you will
not be able to get projects done. If an entire team is focused on emergencies and requests,
nobody is working on projects.
It can be tempting to organize an operations team into three subteams, each focusing on
one source of work or one category of work. Either of these approaches will create silos of
responsibility.Process improvement isbestdonebythepeople involved intheprocess,not
by observers.
To implement our recommended strategy, all members of the team focus on project
workastheirmainpriority.However,teammemberstaketurnsbeingresponsibleforemer-
gency issues as they arise. This responsibility is called oncall . Likewise, team members
take turns being responsible for normal requests from other teams. This responsibility is
called ticket duty .
It is common that oncall duty and ticket duty are scheduled in a rotation. For example,
ateamofeightpeoplemayuseaneight-week cycle.Eachpersonisassignedaweekwhere
heorsheisoncall:expectedtorespondtoalerts,spendinganyremainingtimeonprojects.
Each person is also assigned a different week where he or she is on ticket duty: expected
to focus on triaging and responding to request tickets first, working on other projects only
if there is remaining time. This gives team members six weeks out of the cycle that can be
focused on project work.
Limiting each rotation to a specific person makes for smoother handoffs to the next
shift. In such a case, there are two people doing the handoff rather than a large operations
team meeting. If more than 25 percent of a team needs to be dedicated to ticket duty and
oncall, there is a serious problem with firefighting and a lack of automation.