Upgrading Live Services - The Practice of Cloud System Administration - page 231

the database is modified. There will be no service until replicas are upgraded. You cannot
upgrade the replicas and then change the database schema because any upgraded replica
willfail.Theschema cannotbeupgradedduringtherollingupgrade:newreplicas willfail,
and then at the moment the database schema changes, all the failing replicas will start to
work and the working replicas will start to fail. What chaos! Plus, none of these methods
has a decent way to roll back changes if there is a problem.
One way to deal with this scenario is to use database views . Each view provides a
different abstraction to the same database. A new view is coded for each new software
version. This decouples software upgrades from schema changes. Now when the schema
changes,eachview'scodemustchangetoprovidethesameabstractiontothenewschema.
The change in schema and the upgrade of view code happen atomically, enabling smooth
upgrades. For example, if a field is stored in a new format, one view would store it in the
new format and the other view would convert between the old and new formats. When the
schema changes, the views would reverse roles.
Sadly, this technique is not used very frequently. Views are a rare feature on relational
databases. As of this writing, no key-value store (NoSQL) systems support views. Even
when the feature is available, it isn't always used. In these cases another strategy is re-
quired.
An alternative is to change the database schema by making the change over a period of
two software releases: one after adding any new fields to the database and another before
removing any obsolete fields. We learned this technique from Stephen McHenry and refer
to it as the McHenry Technique. Similar techniques have been called “expand/contract.”
Here are the phases of this technique:
1. The running code reads and writes the old schema, selecting just the fields that it
needs from the table or view. This is the original state.
2. Expand: The schema is modified by adding any new fields, but not removing any
old ones. No code changes are made. If a roll back is needed, it's painless because
the new fields are not being used.
3. Code is modified to use the new schema fields and pushed into production. If a roll
back is needed, it just reverts to to Phase
2. At this time any data conversion can be done while the system is live.
4. Contract: Code that references the old, now unused fields is removed and pushed
into production. If a roll back is needed, it just reverts to Phase 3.
5. Old, now unused, fields are removed from the schema. In the unlikely event that a
roll back is needed at this point, the database would simply revert to Phase 4.