Information Technology Reference
In-Depth Information
Tom: You know my oncall shift ends at 3 PM , right? If you happen to be delayed an
hour...
Rosanne: No such luck. I'm in California and 3 PM your time is when I'll be leaving for
lunch.
A minute after the exercise is over, I receive an email message with a link to a post-ex-
ercise document. I update it with what happened and provide links to the bugs that were
filed. I also think of a few other ways of improving the process and document them, filing
feature requests in our bug database for each of them.
At 2 PM my pager doesn't go off, but I see on my dashboard that there is an outage in
Georgia. Everyone in our internal chat room is talking about it. I'm not too concerned. Our
service runs out of four datacenters around the world, and the system has automatically re-
directed web requests to the other three locations.
The transition is flawless, losing only the queries that were “in flight,” which is well
within our SLA.
A new email appears in my inbox explaining that zombies have invaded Georgia and
are trying to eat the brains of the datacenter technicians there. The zombies have severed
the network connections to the datacenter. No network traffic is going in or out. Lastly, the
emailpointsoutthatthisispartofaDiRTexerciseandnoactualtechnicianshavehadtheir
brains eaten, but the network connections really have been disabled.
[Again, phone rings]
Rosanne: Hi! Having fun yet?
Tom: I'm always having fun. But I guess you mean the Georgia outage?
Rosanne: Yup. Shame about those technicians.
Tom: Well, I know a lot of them and they have big brains. Those zombies will feed for
hours.
Rosanne: Is your service still within SLA?
I look at my dashboard and see that with three datacenters doing the work normally
distributed to four locations the latency has increased slightly, but it is within SLA. The
truth is that I don't need to look at my dashboard because I would have gotten paged if the
latency was unacceptable (or growing at a rate that would reach an unacceptable level if
left unchecked).
Tom: Everything is fine.
Rosanne: Great, because I'm here to proctor another test.
Tom: Isn't a horde of zombies enough?
Search WWH ::




Custom Search