Assessments - The Practice of Cloud System Administration - page 428

• Dashboards exist showing each step's completion time; the lag time of each step is
identified.
• Dashboards exist showing current bottlenecks, backlogs, and idle steps.
• Dashboards show defect and rework counts.
• Capacity planning is performed for the monitoring system and all analysis systems.
• The freshness of the data used to calculate KPIs is measured.
Level 5: Optimizing
• The accuracy of collected data is verified through active testing.
• KPIs are calculated using data that is less than a minute old.
• Dashboards and other analysis displays are based on fresh data.
• Dashboards and other displays load quickly.
• Capacity planning for storage, CPU, and network of the monitoring system is done
with the same sophistication as any major service.
The Unexpectedly Slow Cache
Stack Exchange purchased a product that would accelerate web page delivery to
customers using a globally distributed cache. Most customers deploy this product
and assume it has a “can't lose” benefit.
Beforedeployingit,StackExchangeengineerNickCravercreatedaframework
for measuring end-to-end page load times. The goal was to precisely know how
much improvement was gained both globally and for customers in various geo-
graphic regions.
We were quite surprised to discover that the product degraded performance. It
improved certain aspects but only at the detriment of others, resulting in a net per-
formance loss.
Stack Exchange worked with the vendor to identify the problem. As a result, a
major design error was found and fixed.
If care hadn't been taken to measure performance before and after the change,
Stack Exchange's efforts would have unknowingly made its service slower. One
wondershowmany other customers ofthis product didnosuchmeasurements and
simply assumed performance was improved while in reality it was made worse.