Database Reference
In-Depth Information
Chapter 11
Beyond Aggregation
For many organizations, aggregation and visualization are the end of the
road. Dashboards are created, aggregates are graphed, and reports are
generated. To what end? The answer is usually so that “decision-makers”
can take the “pulse” of whatever system is being monitored. Interpreting this
statement a bit, it would seem to imply that the role of these systems is to
surface information to a human decision-maker so that they can take some
action that affects the system in a desirable way or react to an undesirable
change in that system.
But,thesesystemsarealloperatinginrealtimeandhumansarenotreal-time
creatures. We eat, sleep, and generally do things other than stare at the
continuously updating dashboard. How do we keep up? For mission-critical
things, the solution has usually been to have a large number of humans
working in shifts to keep an eye on the systems.
This works fairly well for relatively small systems, but even reasonably sized
systems like power plants quickly reach the limits of feasibility for the “herd
of humans” approach. This leads to the introduction of automated elements
of the real-time system, such as alarms and automatic shutdown.
These automated systems are the focus of this chapter—helping the human
decision-maker do their job by allowing them to focus on anomalous
behaviors or automating the moment-to-moment decision-making process
altogether. This turns out to be easier said than done, but a number of
approaches have proven successful over time in at least limited capacities.
Automating either the decision process or the process of alerting an operator
to anomalous data requires that the system have some sort of model that
enables it to predict the behavior of the system. The first two sections of this
chapter,“ModelsforReal-TimeData”and“ForecastingwithModels”,discuss
the process of building models, using the concepts introduced in Chapter 9,
“Approximating Streaming Data with Sampling.” This processing of building
a model is called “fitting” and is the essential task of statistical and machine
learning applications. It introduces some of the methods behind classical
modeling, which is usually done “offline” and then applied to the real-time
