Disaster Preparedness - The Practice of Cloud System Administration - page 325

15.5.2 How It Works: IT Operations Arena
When adapting this system to an IT operations organization, the Operations team handles
the operational aspects of the incident—the actual firefighting, as it were. The Logistics
team handles resources, such as people and materials, and makes sure that Operations has
what it needs to do its job. The Planning team is responsible for forecasting situation and
resource needs, and for collecting and displaying information about the incident. The Ad-
min/Finance team handles general administrative support and budgets.
Initially the IC is whoever is first on site. In IT, that is usually the oncall person who
responded to the alert. However, since this person has already been troubleshooting the
alert, it usually makes sense for him or her to sound the alarm and handoff the IC role to
the next qualified responder, continuing as the Operations (Ops) lead. The IC role is trans-
ferred through explicit handoff at shift change or if the IC becomes tired and needs relief.
The key point here is that the IC role is always handed off as part of a thoughtful process,
not automatically as new people show up. If a better-qualified person arrives on scene, the
current IC may decide it's worth the disruption of a handoff to switch ICs.
For small IT incidents, the IC handles all leadership roles on his or her own. However,
as an incident grows larger, more people get involved, either because more co-workers no-
tice the outage or because the IC reaches out to them. As people show up, the IC assigns
them roles, creating subteams as needed. It's worth noting that the IC and the Ops lead
should be made separate individual roles as quickly as possible. The IC has the big-picture
view and the Ops lead is down in the trenches dealing with the incident directly. Trying to
handle both roles simultaneously often results in doing neither role well.
Outagesoflongdurationrequirefrequentstatusupdatestomanagementandotherstake-
holders. By designating a single person to be the Public Information Officer, the IC can
keepfrombeingdistractedbyexecutivesdemandingupdatesorusersasking,“Isitupyet?”
Every status update should end by noting when the next status can be expected. Many or-
ganizations standardize on update frequency. Hourly updates balance executives' need to
know with technical workers' need to focus on the technical issues at hand. Multi-day out-
ages require less frequent updates to avoid repetition.
15.5.3 Incident Action Plan
InICS,anIncidentActionPlan(IAP)iscreatedandcontinuallyrefinedduringtheincident.
This one- to two-page document answers four questions:
• What do we want to do?
• Who is responsible for doing it?
• How do we communicate with each other?