Upgrading Live Services - The Practice of Cloud System Administration - page 238

11.13 Summary
Upgrading the software running in an environment is called a code push. Pushing code re-
quires planning and techniques engineered specifically for upgrading services while they
are running.
Sometimesaservicecanbetakendownforupgradebecauseitruns“behindthescenes.”
More frequently a service is upgraded by taking down parts, upgrading them, and bringing
them back up (i.e., a rolling upgrade). If there are many replicas, one can be designated as
the canary, which is upgraded before the others as a test. Proportional shedding involves
slowly migrating traffic from an old system to a new system, until the old system is receiv-
ing no new traffic.
Flag flips are a technique whereby new features are included in a release but are dis-
abled.Aflagorsoftwaretoggleenablesthefeatureatadesignatedtime.Thismakesiteasy
to revert a change if problems are found: simply turn the toggle off.
Database schema changes on live systems can be difficult to coordinate. It is not pos-
sible to change the software of all clients at the exact same time as the database schema.
Instead, the McHenry Technique is used to decouple the changes. New fields are added,
software is upgraded to use the old and new fields, software is upgraded to use only the
new fields, and any old fields are then removed from the schema.
Whenapushfails,itissometimespossibletorollforwardtothenextreleaseratherthan
reverting code back to the last known good release. Reverting code is risky, and a failed
roll back can be a bigger problem than the original failed push.
Continuous deployment is the process of automating code pushes, automatically paus-
ingordelayingthemwhencertaincriteriaindicatetheriskwouldbetoohighorifbusiness
conditions require a pause. As with continuous integration and continuous delivery, by do-
ing something in an automated manner, frequently, we reduce risk and improve our pro-
cesses.
Exercises
1. Summarize the different techniques for upgrading software in live services.
2. Why is a failed canary process so tragic that development should stop until the
cause is found?
3. Which push techniques in this chapter are in use in your current environment?
4. Which push techniques should your current environment adopt? What would be
their benefits?
5. How would, or could, a dark launch be used in your environment?