Operational Excellence - The Practice of Cloud System Administration - page 403

Operations organizations seem to fall into three broad categories or strata: the great
ones, the ones that want to be great, and the ones that don't even know what great is.
Weestimatethat5to10percentofalloperationsteamsfallintothisfirstcategory.They
know and use the best practices of our industry. Some even invent new ones. The next 25
to 30 percent know that the best practices exist but are struggling to adopt them. The re-
maining super-majority do not even know these best practices exist.
Science fiction writer William Gibson famously said, “The future is already here—it's
just not very evenly distributed.” Likewise, the knowledge of how to be a great system ad-
ministration team is here—it's just not very evenly distributed.
20.2 How to Measure Greatness
Measuring the quality of an operations team is extremely difficult. Other aspects of oper-
ations are easy to measure. For example, size can be measured by counting the number of
team members, the number of services provided, or the dollars spent per year. Scale can be
measured by counting the number of machines, the amount of storage, the total bandwidth
used, and so on. We can measure efficiency using cost ratios.
Alas, quality is not so easy to measure.
Imagine for a moment that we could measure quality. Imagine we had a standard way
to rate the quality of an operations team with a simple value on a scale of 0 to 1000. Also
imagine that we could, possibly by some feat of magic, rank every operations organization
in the world.
If we could do that, we could line all of these organizations up from “best” to “worst.”
The potential for learning would be incredible. We could observe what the top 50 percent
do differently from the bottom 50 percent. Alternatively, a single organization could study
the organizations ranked higher for inspiration on how to improve.
Alas, there is no such single measurement. Operations is just too complex. Therefore
the measurement, or assessment , must reflect that complexity.
Assessments sound a lot like the grades we received in school, but the concept is very
different.Astudentassessmentevaluatesanindividualstudent'slearningandperformance.
Grades assess learning, but they also incorporate attendance, participation, and effort. An
assessment is more focused.
An assessment of a service is an evaluation based on specific criteria related to process
maturity. It is not an evaluation of whether the service is popular, has high availability, or
is fast. Not all services need to be popular, highly available, or fast. In contrast, all services
need good processes to achieve whatever goals they do have. Therefore we assess process
because good processes are a roadmap to success.