Design Patterns for Scaling - The Practice of Cloud System Administration - page 96

5.1.3 Measure Results
Scaling solutions must be evaluated using evidence, meaning data collected from a real
system. Take measurements, try a solution, and repeat the same measurements to see the
effect. If the effect is minimal or negative, the solution is not deployed.
For example, if performance is slow and measurements indicate that a cache hit rate is
very low, the cache is probably too small. In such a case, we would measure performance,
resize thecache, andthenmeasure performance again. While thecache maybeperforming
better with such an adjustment, the overall effect on the system may not be a significant
improvement or may not be big enough to justify the cost of the additional RAM required.
If we do not measure before and after a change is made, we cannot be sure whether our
changes were actually effective. Making changes without measurement would be system
administrationbyluckatbest,andbyegoatworst.Itisoftentemptingtorushaheadwitha
solution and measure only after the change is made. This is as bad as not measuring at all,
because there is no baseline for comparison.
Whilepastexperienceshouldinformandguideus,wemustresistthetemptationtoskip
the scientific process of using measurement and analysis to guide our decisions. Distrib-
utedsystemsarealwaystoolargeforanyonepersonto“justknow”therightthingtodo.A
hunch or guess by an experienced system administrator should be trumped by the recom-
mendation of a more junior person who has taken the time to perform scientific analysis:
set up data collection mechanisms, take measurements, verify a hypothesis about what is
wrong, test a theory of what might fix it, and analyze the results.
5.1.4 Be Proactive
The best time to fix a bottleneck is before it becomes a problem. Ideally, the fix will arrive
justintime,immediatelybeforethispointisreached.Ifwefixaproblemtoosoon,wemay
be wasting effort on a problem that never arises—effort that could have been better spent
elsewhere. If we begin to design and implement a solution too late, the problem will arise
before the solution is deployed. If we wait much too long, the problem will surprise us,
catching us off guard, and we will be “firefighting” to find a solution. Engineering takes
time and doing it in a rushed fashion leads to mistakes and more problems. We want the
Goldilocks solution: not too early, not too late, just right.
Every system has a bottleneck or constraint. This is not a bad thing. Constraints are in-
herent inall systems. Aconstraint dictates the rate at which processing can happen, orhow
much work can flow through the process. If the current rate is sufficient, the bottleneck is
notaproblem.Inotherwords,theconstraintbecomesaproblemonlyifitactuallyhampers
the system's ability to achieve its goal.