Upgrading Live Services - The Practice of Cloud System Administration - page 230

overload the system. Therefore, the JavaScript is inserted only 1 out of every 100
times. The flag would be set to 0 to disable collection, or a value specifying the
sampling percentage.
• Differentiated Services: Sometimes there is a need to enable different services for
different users. A good flag system can enable paid customers to see different fea-
tures than unpaid users see. Many membership levels can be implemented by asso-
ciating a set of flags with each one.
Sometimes a flag is used to disable everything to do with a new feature; at other times it
just obscures its visibility. Suppose a web site is going to add auto-completion to an input
field.ThisfeaturewillrequirechangesintheHTMLthatisgeneratedfortheinputprompt,
the addition of a new API call to the database for partial queries, and possibly other code.
The flag could disable everything to do with the feature or it may simply control whether
the feature appears in the UI.
Case Study: Facebook Chat's Dark Launch
In 2011 Facebook's Chuck Rossi gave a presentation titled “Pushing Millions of
Lines of Code Five Days a Week” ( Rossi 2011 ) that described an in-house tool
called “Gatekeeper” that manages dark launches : code launched into production
with no user-visible component. Rossi's talk revealed that at any given moment
on Facebook.com, there is already the code for every major thing Facebook is go-
ing to launch in the next six months and beyond. Gate-keeper permits fine-grained
control over which features are revealed to which users. Some filters are obvi-
ous, such as country, age, and datacenter. Others are unexpected, including one to
exclude known employees of certain media outlets such as TechCrunch ( Siegler
2011 ) . (Also see the related case study in Section 18.4 .)
11.8 Live Schema Changes
Sometimes a new software release expects a different database schema from the previous
release. If the service could withstand downtime, one would bring down the service, up-
grade the software and change the database schema, and then restart the service. However,
downtime is almost always unacceptable in a web environment.
In a typical web environment there are many web server frontends, or replicas, that all
talktothesamedatabaseserver.Theoldersoftwarereleaseisunabletounderstandthenew
databaseschemaandwillmalfunctionorcrash.Thenewersoftwarereleaseisunabletoun-
derstand the old database schema and will also malfunction. Therefore you cannot change
thedatabaseschemaandthendothesoftwareupgrade:theolderreplicaswillfailassoonas