Database Reference
In-Depth Information
output = new
SystemStream("kafka",config.get("count.output",
"stats"));
}
}
Multi-Resolution Time-Series Aggregation
Systems-monitoring applications have long relied on so-called round-robin
databasesforstorageoftime-seriesdata.Thesesimpledatabasesimplement
a circular buffer that stores some metric: CPU load every second, a count of
a number of events, and so on.
Round-robin databases, being essentially circular buffers, can only hold a
fixed number of data points. As a result, monitoring tools typically maintain
several such databases for each metric with larger and larger time intervals.
Using this method, very high-resolution data (say one-second intervals) is
available for a short time period, such as 24 hours. For a single metric,
that would be a circular buffer of 86,400 entries. The next largest buffer
would aggregate information at the minute level. Using the same sized
buffer would allow for the storage of 60 days' worth of data at the minute
level. Moving up to hourly aggregation would allow for the storage of more
than nine years' worth of data.
Quantization Framework
Rather than implement a round-robin database with a fixed amount of
storage, you can use a NoSQL storage back end with key expiration serving
the same function as the round-robin database. While not as compact, this
approach is easy to integrate into front-end services and maintains bounded
memory usage provided the number of possible metrics remains in a fairly
steady state.
A multi-resolution counting framework relies on two pieces. The first is a
generic piece of code that defines the aggregation time frame and retention
time. The second part defines aggregation operations for a specific database
implementation. Although it is possible to make this piece generic, doing so
requires the interface to either only implement the lowest common set of
featuresorrequiremassiveamountsofworkforback-enddatabasesthatdid
not implement some specific feature.
Search WWH ::




Custom Search