Oncall - The Practice of Cloud System Administration - page 291

• One-Way and Two-Way Pagers: Hand-held devices that receive text messages
via terrestrial broadcasts. Two-way pagers permit sending a reply to acknowledge
that the message was received.
• SMS or Text Message to a Mobile Phone: Sending a text or SMS message to a
person's mobile phone is convenient because most people already carry a mobile
phone. In some countries, pagers are significantly more reliable than SMS; in oth-
ers, the reverse is true. If you are creating an alerting system for co-workers in an-
other country, do not assume that what works well for you will be viable else-
where. Local people should test both.
• Smart Phone App: Smart phone apps are able to display additional information
beyond a short text message. However, they often depend on Internet connectivity,
which may not always be available.
• Voice Call: A voice synthesizer and other software is used to call a person's phone
and talk to him or her, asking the person to press a button to acknowledge the mes-
sage (otherwise, the escalation list will be activated).
• Chat Room Bot: A chat room bot is a software robot that sits in the team's chat
room and announces any alerts. This is a useful way to keep the entire team en-
gaged and ready to help the oncall person if needed.
• Alerting Dashboard: The alerting dashboard is a web page that shows the history
of alerts sent. It provides useful context information.
• Email: Email should never be the only way the oncall person is alerted. Sitting at
your computer watching your inbox is a terrible use of your time. Nevertheless, it
is useful to have every alert emailed to the oncall person as a backup method. This
way the full message is received; SMS truncates messages to 160 characters.
At least two methods should be used to make sure the message gets through. A pager ser-
vicemighthaveanoutage.AsleepingoncallpersonmightrequireanSMStoneandavoice
call to wake him or her. One method should communicate the complete, non-truncated,
message to a stable storage medium. Email works well for this. Lastly, chat room bots and
othermethodsshouldalsobedeployed,especiallygiventhatthisstrategyenablestheentire
team to stay aware of issues as they happen. They may seem like novelties at first but they
quickly become indispensable.
14.1.8 After-Hours Maintenance Coordination
Sometimes maintenance must be done after hours. For example, another team may do
maintenance that requires your participation, such as failing over services so they are not
affected by scheduled outages, or being around to verify status, and so on.