Disaster Preparedness - The Practice of Cloud System Administration - page 318

Organization
Two subteams are needed to make the event a success. The tech team designs the global
testsandevaluatesteamtestplans.Thetestsaregradedonquality,impact,andrisk.During
the event this team is responsible for causing the global outages, monitoring them, and
making sure things don't go awry. This team also handles unforeseen issues caused by the
tests.
The coordination team is responsible for planning, scheduling, budgets, and execution.
It works with the tech team to prevent conflicting tests. Coordination team members verify
that all prep work is complete. They also are in charge of communication both with man-
agement and corporate-wide.
Duringtheevent,bothsubteamsresideinthecommandcenter,whichoperatesasakind
of “Mission Control.” Due to the nature of the tests, the command center must be a phys-
ical location, as relying on virtual meeting spaces is too risky. For large organizations, this
involves a considerable amount of travel.
Inthecommandcenter,individualscallproctorstotellthemtostarttheirtests.Testcon-
trollers deal with unexpected issues as they come up and communicate globally about the
status of the event. They monitor the progress of the tests as well as the effects of the tests
on the technical systems that should not be affected. Test controllers can halt all tests if
something goes massively wrong.
Oneofthemostvaluablerolesinthecommandcenteristhestoryteller.Onepersoncon-
coctsandnarratesthedisaster.Heorsheshouldcreateastorythatisjustunrealisticenough
that people know that it is a test. For example, the story might involve a zombie attack, an
evil genius who has put everyone at headquarters under hypnosis, or an errant fortune-tell-
er with mystical powers. Daily updates that move the story forward should be distributed
by email or old-time radio “broadcasts.” The last bulletin should resolve the story and let
everyone know that the test is over, the zombies have been defeated, or whatever.
The story teller helps counteract the drudgery of the test. If this person does a good job,
people might actually look forward to next year's event.