Designing for Operations - The Practice of Cloud System Administration - page 35

Itisoftenunclearwhatvendorsmeanby“hot-pluggable”and“hot-swappable.”Directly
test the system to understand the ramifications of the process and to see how application-
level software responds.
2.1.9 Toggles for Individual Features
A configuration setting (a toggle ) should be present to enable or disable each new feature.
This allows roll-outs of new software releases to be independent of when the new feature
is made available to users. For example, if a new feature is to appear on the site precisely
at noon on Wednesday, it is very difficult to coordinate a new binary “push” exactly at that
time. However, if each new feature can be individually enabled, the software can be de-
ployed early and the feature can be enabled via changing a configuration setting at the de-
sired time. This is often called a flag flip .
Thisapproachisalsousefulfordealingwithnewfeaturesthatcauseproblems.Itiseasi-
er to disable the individual feature via a flag flip than to roll back to the previous binary.
More sophisticated toggles can be enabled for particular groups of users. A feature may
be enabled for a small group of trusted testers who receive early access. Once it is valid-
ated, the toggle can enable the feature for all users, perhaps by enabling it for successively
larger groups.
See Section 11.7 for more details.
2.1.10 Graceful Degradation
Graceful degradation means software acts differently when it is becoming overloaded or
when systems it depends on are down. For example, a web site might have two user inter-
faces: one is rich and full of images, while the other is lightweight and all text. Normally
users receive the rich interface. However, if the system is overloaded or at risk of hitting
bandwidth limits, it switches to the lightweight mode.
Graceful degradation also requires the software toact smartly duringoutages. Aservice
may become read-only if the database stops accepting writes (a common administrative
defense when corruption is detected). If a database becomes completely inaccessible, the
software works from its cache so that users see partial results rather than an error message.
When a service can no longer access services it depends on, related features may disap-
pear rather than just displaying a broken web page or a “404 page not found” error.
Even small sites have learned that it is better to put up a temporary web server that dis-
plays the same “under construction” page no matter what the query, than to have the users
receive no service at all. There are simple web server software packages made just for this
situation.