Monitoring Architecture and Practice - The Practice of Cloud System Administration - page 347

High-Speed Network Counters
The frequency and potential rate of change should determine the size of the
counter. If the counter that stores the number of bits transmitted on a 10 Gbps in-
terface is 32 bits long, it will roll over to zero after 4,294,967,296 bits have been
transmitted.Iftheinterfaceisrunningatabout50percentcapacity,thecounterwill
rolloveraboutevery8.5seconds.Itisnotpossibletocorrectlysynthesizethemet-
ric for traffic across this interface unless you collect data faster than the counter
rolls over. Therefore a counter that changes this quickly requires a 64-bit counter.
A counter is a measurement that only increases—for example, a count of the number
of API calls received by a service or the count of the number of packets transmitted on
a network interface. A counter does not decrease or run backward. You can't unsend a
packet or deny that an API call was received after the fact. However, counters do reset
to zero in two circumstances. Unless a counter's value is in persistent storage, the value
is reset to zero on reboot or restart. Also, like an automobile odometer that rolls over at
99,999, a counter loops back to zero when it reaches its maximum value. Typically coun-
ters are 16-, 32-, or 64-bit integers that have maximum values of 65,535, 4,294,967,295,
and 18,446,744,073,709,551,615, respectively (i.e., 2 n - 1, where n is the number of bits).
Doing math on counter readings is rather complex. In our previous example, we simply
subtracted adjacent counter values to learn the delta. However, if the second reading is less
than the first, special care must be taken.
If the counter looped around to zero, the actual delta between two readings includes the
count up to the maximum value plus the newer counter value: (maxvalue - R n ) + R n +1 .
However, if the counter was reset to zero due to a restart, then the exact delta value is un-
obtainable because we do not know the value of the counter when the restart occurred. We
can make estimates using a little high school calculus involving the past rate and how long
ago the reset occured.
Determining whether the counter reset to zero due to a restart or reaching the maximum
value is another heuristic. If the last reading was near the maximum value, we can assume
the counter looped around to zero. We can improve the accuracy of this guess, again by
applying a little calculus.
Such great complexity still results in a margin of error and risks introducing a loss of
accuracy if either heuristic produces incorrect results. A much simpler approach is to ig-
nore the problem. For 64-bit counters, wrap-arounds are rare or nonexistent. For counter
resets, if the monitoring frequency is fast enough, the margin of error will be very small.
Usually the desired end result is not the count itself, but a rate. The calculation of the rate