Monitoring Architecture and Practice - The Practice of Cloud System Administration - page 346

17.1.3 Rate versus Capability Monitoring
Eventfrequencydetermineswhattomonitor.Onewantstoknowthatacustomer can make
purchases and is making purchases. The rate at which purchases happen determines which
of these to monitor. If one or two purchases are made each day, monitoring whether cus-
tomers can make purchases is important because it happens so infrequently that we need
to verify that the purchasing flow hasn't somehow broken. If thousands of purchases per
minute are made, then we generally know purchases can be made and it is more important
to measure the rate and trends.
In short, rate metrics are more important when event frequency is high and there are
smooth,predictabletrends.Whenthereisloweventfrequencyoranunevenrate,capability
metrics are more important.
That said, never collect rates directly. Rates are lossy. Given two counts and their time
delta, we can compute the rate. Given two rates and their time delta, we can only surmise
the count—and then only if the two measurements were collected at precise intervals that
are in sync with the rate's denominator.
For example, if we are measuring the count of how many API calls have been received,
wemightcollectametricof10,000at12:10:00and16,000at12:15:00.Wecouldthencal-
culate the rate by dividing the delta of the counts by the time delta: (16, 000-10, 000)/300,
or 20 API calls per second. Suppose that second polling event was skipped, perhaps due
to a brief network problem, the next polling event would be at 12:20:00 and might collect
the measurement of21,000.We can still calculate the rate: (21,000-10,000)/600,or18.2
API calls per second. However, if we were collecting rates, that network blip would mean
we would not be able to estimate what happened during the missing 5 minutes. We could
take the average of the two adjacent rates, but if during that time there was a large spike,
we would not know.
17.1.4 Gauges versus Counters
Some measurements are gauges, while others are counters.
A gauge value is an amount that varies. It is analogous to a real-world gauge like one
thatindicatesbarometricpressure.Examplesincludeanindicatorofhowmuchunuseddisk
space is available, a temperature reading, and the number of active processes on a system.
A gauge's value varies, going up and down as whatever it measures increases or decreases.