Designing for Operations - The Practice of Cloud System Administration - page 41

1. Announce the impending failover to the db-team and manager-team mailing lists.
2. Verify the hot spare has at least 10 terabytes of free disk space.
3. Verify these dependencies are all operating within parameters: (link to server con-
trol panel) (link to data feed control panel).
4. Perform the failover using the system failover procedure (link).
5. Verify that these dependencies have successfully switched to the hot spare and are
operating correctly.
6. Reply-all to the previously sent email regarding the operation's success or failure.
Documenting the basic operational procedures for a service cannot be the sole responsib-
ility of either the development or operations team. Instead, it needs to be a collaborative
effort, with the operations team ensuring that all the operational scenarios they can foresee
are addressed, and the developers ensuring that the documentation covers all error situ-
ations that can arise in their code, and how and when to use the supporting tools and pro-
cedures.
Documentation is a stepping stone to automation. Processes may change frequently
when they are new. As they solidify, you can identify good candidates for automation (see
Chapter12 ) .Writingdocumentation alsohelpsyouunderstandwhatcanbeautomated eas-
ily and what will be more challenging to automate, because documenting things means ex-
plaining the steps in detail.
2.2 Implementing Design for Operations
Features designed for operations need to be implemented by someone; they are not magic-
ally present in software. For any given project, you will find that the software may have
none, some, or all of the features listed in this chapter. There are four main ways that you
can get these features into software:
• Build them in from the beginning.
• Request features as they are identified.
• Write the features yourself.
• Work with a third-party vendor.
2.2.1 Build Features in from the Beginning
In this instance, a savvy development and operations team has built the features into the
product you are using to run your service. It is extremely rare that you will encounter this
scenario outside of large, experienced shops like Google, Facebook, or Yahoo.