Capacity Planning - The Practice of Cloud System Administration - page 384

• Modify an existing transaction to behave differently when the dark launch toggle is
“on.” In this case, photo upload will be modified to run the sample photo editing
transaction 25 percent of the time that uploads occur, feeding the uploaded photo
into the editing transaction.
The 25 percent value is only an example—it could be any percentage. It simply represents
a known quantity that can be calculated at a later time based on transaction data. Starting
withamoreconservativenumbersuchas5percentislikelytobeagoodideawithanactual
complex service. Use the resource regression analysis technique to see which kind of re-
sourcecostsareassociatedwiththenewlaunchfeature.Thiswillgiveyouafirstpassatca-
pacity planning for the new feature launch, based on actual production environment usage.
Adjust your capacity accordingly, and then continue the dark launch, fixing any bugs that
are found and adjusting capacity as needed. Gradually increase the percentage of sample
dark launch transactions until you reach a level where real usage is likely to occur. Be sure
to go beyond that by some percentage to give yourself headroom.
Finally the dark launch is turned off and the feature is turned on for the customers.
Withtheappropriate toggles,thiscanallbedonewithoutrollinganewrelease. Inpractice,
enoughbugfixinghappensduringadarklaunchthatyouwillalmostcertainlyupgradenew
releases during the dark launch period. However, all of these bugs should be invisible to
your customer community, and your actual launch will be the better for having done a dark
launch.
Case Study: Facebook Chat's Dark Launch
Theterm“darklaunch”wascoinedin2008whenFacebookrevealedthetechnique
was used to launch Facebook Chat. The launch raised an important issue: how to
gofromzeroto70millionusersovernightwithoutscalingissues.Anoutagewould
be highly visible. Long before the feature was visible to users, Facebook pages
were programmed to make connections to the chat servers, query for presence in-
formation, and simulate message sends without a single UI element drawn on the
page( Letuchy2008 ) .ThisgaveFacebookanopportunitytofindandfixanyissues
ahead of time. If you were a Facebook user back then, you had no idea your web
browser was sending simulated chat messages but the testing you provided was
greatly appreciated. ( Section 11.7 has a related case study.)