Monitoring Architecture and Practice - The Practice of Cloud System Administration - page 350

17.2.3 Server Component versus Agent versus Poller
Collectors can work in several different ways. The cleanest way to implement whitebox
monitoring is to have the running server gather its own measurements and handle pushing
or pulling the results to the collector. The software that provides these functions can be
put into a library for convenient use by any program. For example, every time an API call
is received, the software might call metric.increment('api-calls-count')
to increment the count called api-calls-count . To collect how much bandwidth is
being used in a video server, after each write to the network the software might call
metric.add('bandwidth-out', x) , where x is the amount of data sent. The lib-
rarywouldmaintainarunningtotalunderthename bandwidth-out .Functionslikethis
are called throughout the program to count, total, or record measurements.
If the collection is done by push, the library spawns a thread that wakes up periodically
to push the metrics to the monitoring system. Pull is implemented by the library spawning
athreadthatlistenstoaspecific TCPportforpollingeventsandreplieswithsomeorallof
the metrics. If the server already processes HTTP requests, it can simply attach the library
to a particular path or route. For example, HTTP requests for /monitor would call the
library, which would then reply to the request with a JSON representation of the metrics.
Sadly, we can't always modify software to collect metrics this way. In that case we run
software on the machine that collects data by reading logs, status APIs, or system para-
meters and pulls or pushes the information. Such software is called an agent . For example,
scollector is a software agent that runs on a Linux machine and calls any number of plug-
ins to gather metrics, which are then pushed to the Bosun monitoring system. Plug-ins are
available that collect performance information about the operating system, disk systems,
and many popular open source systems such as MySQL and HBase.
Sometimes we cannot modify software or install software on the machine itself. For ex-
ample, hardware devices such as network switches, storage area network (SAN) hardware,
and uninterruptible power supplies (UPSs) do not permit user-installed software. In these
cases software is run on one machine that polls the other devices, collects the information,
and transmits it via push or pull to the monitoring system. This type of software is called a
poller or a monitoring proxy .