Monitoring Architecture and Practice - The Practice of Cloud System Administration - page 348

fortheentireintervalwillnotbedramaticallyaffected.Forexample,ifthereare10counter
readings, 9 deltas are produced. If a delta is negative, it indicates there was a counter reset.
Simply throw away any negative deltas and calculate the rate with the remaining data.
Java Counters
SystemswrittenintheJavaprogramminglanguageusecountersthataresignedin-
tegers. Signed integers roll over to negative numbers and have a maximum value
that is approximately half their signed counterparts.
17.2 Collection
Once we have a measurement, we must transmit it to the storage system. Metrics are col-
lectedfrommanyplacesandbroughttoastoragesystemsothattheymaybeanalyzed.The
metric's identity must be preserved through any and all transmissions.
There are hundreds of ways to collect metrics. Most fall into one of two camps: push or
pull. Irrespective of the choice of a push or pull mechanism, there is also a choice of which
protocolsareusedfordatacollection.Anotheraspectofdatacollectioniswhethertheserv-
er itself that communicates directly with the collector, or whether an external agent acts
as an intermediary between the server and the collector. A monitoring system may have
a central collector or regional collectors that consolidate data before passing it back to a
central system.
17.2.1 Push versus Pull
Push meansthesensorthattookthemeasurementtransmitsittothecollectionmechanism.
Pull means an agent polls the object being monitored and requests the data and stores it.
Four myths in monitoring are that pull doesn't scale, that push is horrible and shouldn't
be used, that pull is horrible and shouldn't be used, and that push doesn't scale. Dispelling
these myths is easy.
The ability to scale is a matter of implementation, not push versus pull. Monitoring
10,000 items at least once every 5 minutes requires 33 connections per second if spread
evenly. A web server can handle millions of inbound TCP connections. Outbound connec-
tions are no different. A monitoring system has the benefit that it can make certain optim-
izations. It can transmit multiple measurements over each TCP connection. It can simply
create a long-lived TCP connection for every machine. Scaling is a concern, but not an in-
surmountable obstacle.
Whether push or pull is better is a matter of application. Pull is better for synthesized
measurements, such as counters. It is also better for direct measurements when change is