Assessments - The Practice of Cloud System Administration - page 430

Level 2: Repeatable
• CP metrics are collected on demand, or only when needed.
• The process for collecting CP metrics is written and repeatable.
• Load testing is done occasionally, perhaps when a service is new.
• Inventory of all systems is accurate, possibly due to manual effort.
Level 3: Defined
• CP metrics are automatically collected.
• Capacity required for a certain amount of growth is well defined.
• There is a dedicated CP person on the team.
• CP requirements are defined at a subsystem level.
• Load testing is triggered by major software and hardware changes.
• Inventory is updated as part of capacity changes.
• The amount of headroom needed to survive typical surges is defined.
Level 4: Managed
• CP metrics are collected continuously (daily/weekly instead of monthly or
quarterly).
• Additional capacity is gained automatically, with human approval.
• Performance regressions are detected during testing, involving CP if performance
regression will survive into production (i.e., it is not a bug).
• Dashboards include CP information.
• Changes in correlation are automatically detected and raise a ticket for CP to verify
and adjust relationships between core drivers and resource units.
• Unexpected increases in demand are automatically detected using MACD metrics
or similar technique, which generates a ticket for the CP person or team.
• The amount of headroom in the system is monitored.
Level 5: Optimizing
• Past CP projections are compared with actual results.
• Load testing is done as part of a continuous test environment.
• The team employs a statistician.
• Additional capacity is gained automatically.