Monitoring Architecture and Practice - The Practice of Cloud System Administration - page 345

testing includes monitoring that attempts to do multi-step processes as a user, such as veri-
fying that the purchasing process is working.
It is tempting to get a particular web page and verify that the HTML received is correct.
However, then the monitoring system must be updated anytime the web site changes. An
alternative is to verify that the page contains a particular message. For example, a search
engine might periodically query for the name of a particular celebrity and verify that the
nameoftheperson'sfanclubismentionedintheHTML.Themeasurementcommunicated
to the monitoring system would be 1 or 0, representing whether the string was or wasn't
found.
Blackbox monitoring is often highly dependent on perspective. Monitoring might be
done from the machine itself, from within the datacenter or other failure domain, or from
different places around the world. Each of these tests answers different questions about the
system.
A whitebox measurement has the benefit of internal knowledge because it is a lower
level of abstraction. For example, such a measure might monitor raw counters of the num-
ber of times a particular API call was made, the number of outstanding items waiting in
a queue, or latency information of internal processes. Such measurements may disappear
or change meaning as the system's internals change. In whitebox monitoring, perspective
is less important. Usually such measurements are done as directly as possible to be more
efficient and to prevent data loss that could occur by adding distance.
17.1.2 Direct versus Synthesized Measurements
Some measurements are direct, whereas others are synthesized. For example, if every time
a purchase is made the system sends a metric of the total money collected to the monitor-
ing system, this is a direct measurement. Alternatively, if every 5 minutes the monitoring
system tallies the total money collected so far, this is a synthesized metric. If all we have is
a synthesized metric, we cannot back-calculate the individual amounts that created it.
Whether direct measurements are available is generally a function of frequency of
change.Inane-commercesitewhereoneortwopurchasesaremadeeachday,directmeas-
urement is possible. By comparison, for a busy e-commerce site such as eBay or Amazon,
the monitoring system could not keep up with the number of purchases made if each one
resulted in a metric. In this case the synthesized metric of total revenue collected is all that
is possible. Exact counts would be the job of the accounting and billing system.
A more obvious example is measuring how much disk space is in use versus being told
about every block allocation and deallocation. There can be millions of disk operations
each minute. Collecting the details of each operation just to arrive at a metric of how much
disk space is in use would be a waste of effort.