Oncall - The Practice of Cloud System Administration - page 287

six. With a small team, a daily schedule might mean being oncall every six days,
never having a complete week to take a vacation.
• Split Days: On a given day multiple people are oncall, each one responsible for a
different part of the day or shift. For example, a two-shift schedule might involve
two 12-hour shifts per day. One person works 9 AM to 9 PM and another is oncall
for the overnight. This way, if an alert happens in each shift, someone is always
able to sleep. A three-shift schedule might be 8 hours each: 9 AM to 5 PM , 5 PM to 1
AM , and 1 AM to 9 AM .
• Follow the Sun: Members of the operations team live in different time zones, and
each is oncall for the hours that he or she would normally be awake (sun-light
hours). If the team resides in California and Dublin, a shift change at 10 AM and 10
PM California time means all members have some responsibilities during office
hours and sleeping hours, plus there are enough overlap hours for inter-team com-
munication. A team split between many time zones may have three or four shifts
per day.
Many variations are also possible. Some teams prefer half-weeks instead of full weeks.
Followthesuncanbedonewithwithtwo,three,orfourshiftsperdaydependingonwhere
people are located.
Alert Frequency
With so many variations, it can be difficult to decide which to use. Develop a strategy
where the frequency and duration of oncall shifts are determined by how frequent alerts
are. People do their best oncall work when they are not overloaded and have not been kept
awake for days on end. An oncall system improves operations when each alert receives
follow-up work such as causal analysis. Before someone goes oncall again, that person
should have had enough time to complete all follow-up work.
For example, if alerts are extremely rare, occurring less than once a week, a single per-
son being oncall for 7 days at a time is reasonable. If there are three alerts each day, two or
three shifts gives people time to sleep between alerts and do follow-up tasks. If follow-up
work is extensive, half-week rotations of two shifts may be required.
If there is a significantly high alert ratio, more complex schemes are needed. For ex-
ample, in some schedules, two people are oncall: one receives the alert if the other is busy,
plus two secondaries shadow the primaries. Some systems divide the work geographically,
with a different rotation scheme and schedule for each major region of the world.