Capacity Planning - The Practice of Cloud System Administration - page 383

18.4 Launching New Services
Now that we understand how to plan for capacity on an existing system, let's talk about
launchingnewservices.Launchinganewserviceisdifficultandriskybecauseyouhaveno
prior experience or service metrics to plan the initial capacity required. For large services,
testing may be unreliable, as there may be insufficient capacity on a staging environment
for a true test.
Adding to this risk, launch day is often when the service will have the most media scru-
tiny.Ifthesystemrunsoutofcapacityandbecomesunreliable,themediawillhavenothing
to write about except what a terrible launch it was. This makes a bad first impression on
customers.
One way to mitigate this risk is to find a way to have a slow ramp-up in the number of
active users. For example, enabling a product but not announcing it (a soft launch ) gives
engineers time to find and fix problems as they happen. However, if additional resources
are needed and will require weeks of lead time, only the slowest of ramp-ups will help.
Relying on a slow ramp-up is not an option for well-known companies. Google and
Facebook haven't had a slow ramp-up on a new service, even with a soft launch, for years.
Any new service is immediately flooded with new users.
Thereforetheabilitytodoaccuratecapacityplanningonnewserviceshasbecomeahighly
desired skill.
Fortunately, there is a technique that can be used to bridge the knowledge gap, and that
technique is the dark launch . Facebook used a dark launch of its chat messaging system
to ensure that the company would bring a reliable system to its user base.
In a dark launch, the new feature is released into production with simulated traffic, in
effecttreatingtheproductionenvironmentasawell-controlledtestbedforthenewfeature's
resource needs. No user-visible information is created—service agents exercise a feature
silentlyfromtheuserperspective,butemployrealuseractivitytotriggersimulatedactivity
for the new feature.
For example, suppose we wanted to add a photo editing capability to our photo sharing
site from the earlier example. To do a dark launch, we might do the following:
• Create a software toggle for the photo editing feature: on or off. (Software toggles
were described in Section 2.1.9 .)
• Create a dark launch toggle for the photo editing feature: on or off.
• Create a sample photo editing transaction that is saved to a dummy area. The cus-
tomer's photo is not changed, but behind the scenes the photo is edited and a modi-
fied version saved elsewhere.