Operations in a Distributed World - The Practice of Cloud System Administration - page 148

7.1.2 Change versus Stability
There is a tension between the desire for stability and the desire for change. Operations
teams tend to favor stability; developers desire change. Consider how each group is evalu-
ated during end-of-the-year performance reviews. A developer is praised for writing code
that makes it into production. Changes that result in a tangible difference to the service
are rewarded above any other accomplishment. Therefore, developers want new releases
pushedintoproductionoften.Operations,incontrast,isrewardedforachievingcompliance
with SLAs, most of which relate to uptime. Therefore stability is the priority.
A system starts at a baseline of stability. A change is then made. All changes have some
kind of a destabilizing effect. Eventually the system becomes stable again, usually through
some kind of intervention. This is called the change-instability cycle .
All software roll-outs affect stability. A change may introduce bugs, which are fixed
throughworkaroundsandnewsoftwarereleases.Areleasethatintroducesnonewbugsstill
creates a destabilizing effect due to the process of shifting workloads away from machines
about to be upgraded. Non-software changes also have a destabilizing effect. A network
changemaymakethelocalnetworklessstablewhilethechangepropagatesthroughoutthe
network.
Becauseofthetensionbetweentheoperationaldesireforstabilityandthedeveloperde-
sire for change, there must be mechanisms to reach a balance.
One strategy is to prioritize work that improves stability over work that adds new fea-
tures. For example, bug fixes would have a higher priority than feature requests. With this
approach, a major release introduces many new features, the next few releases focus on
fixing bugs, and then a new major release starts the cycle over again. If engineering man-
agement is pressured to focus on new features and neglect bug fixes, the result is a system
that slowly destabilizes until it spins out of control.
Another strategy is to align the goals of developers and operational staff. Both parties
become responsible forSLA compliance as well as the velocity (rate ofchange) ofthe sys-
tem.BothhaveacomponentoftheirannualreviewthatistiedtoSLAcomplianceandboth
have a portion tied to the on-time delivery of new features.
Organizationsthathavebeenthemostsuccessfulataligninggoalslikethishaverestruc-
tured themselves so that developers and operations work as one team. This is the premise
of the DevOps movement, which will be described in Chapter 8 .
Anotherstrategyistobudgettimeforstabilityimprovementsandtimefornewfeatures.
Software engineering organizations usually have a way to estimate the size of a software
request or the amount of time it is expected to take to complete. Each new release has a
certain size or time budget; within that budget a certain amount of stability-improvement