Designing for Operations - The Practice of Cloud System Administration - page 28

• Hot swaps
• Toggles for individual features
• Graceful degradation
• Access controls and rate limits
• Data import controls
• Monitoring
• Auditing
• Debug instrumentation
• Exception collection
Features like configuration and backups/restores make typical operational tasks possible.
Featureslikequeuedrainingandtogglesforindividualfeaturesallowsuchtaskstobedone
seamlessly. Many of these features create the introspection required to debug, tune, and re-
pair large systems, as previously discussed in Section 1.1 .
Typically, developers and managers don't think of these issues and they are often left
off of lists of requirements. At best, they are after-thoughts or it is assumed that the opera-
tions team will “figure something out” through improvisation. The truth is that these tasks
are important and cannot be done well or at all without specific features. In the worst case,
systems are designed in ways that work against the ability to “improvise” a solution.
Ratherthantheterm“operationalrequirements,”someorganizationsusetheterm“non-
functional requirements.” We consider this term misleading. While these features are not
directly responsible for the function of the application or service, the term “non-function-
al” implies that these features do not have a function. A service cannot exist without the
support of these features; they are essential.
The remainder of this chapter discusses these operational aspects and features that en-
able them. Many of these will seem obvious to someone with operational experience. Yet,
eachofthemappearsonthislistbecausewehaveobservedatleastonesystemthatsuffered
from its omission.
2.1.1 Configuration
The system must be configurable by automated means. This includes initial configuration
as well as changes made later. It must be possible to perform the following tasks:
• Make a backup of a configuration and restore it
• View the difference between one archived copy and another revision
• Archive the running configuration without taking the system down