Database Reference
In-Depth Information
Statistical Modeling
Now that your large volumes and different formats of machine data have
been normalized and transformed, you can start performing some meaning-
ful analysis. Even without considering the different log types, volume can
make statistical analysis prohibitive. In a Hadoop context, even though it's
possible to deal with high data volumes and varieties efficiently, it's difficult
to program statistical algorithms. BigInsights makes life easier here because
it has a toolkit for machine learning and deep statistical analysis. The MDA
leverages some of these statistical algorithms to help reveal valuable infor-
mation that's locked in your machine data. There are currently two statistical
models available for the MDA: Frequent Subsequence Identification and Sig-
nificance Analysis. (Our lawyers don't like us hinting into the future, but
imagine an MDA that has more statistical models at your disposal—sooner
than later.) Both of these statistical models provide output that can easily be
visualized and graphed using BigSheets.
The Frequent Subsequence Identification model shows which sequences of
events happen most frequently across different sessions. A number of interest-
ing patterns can be revealed with this analysis, which enables proactive admin-
istration to prevent future problems. For example, you can identify the series of
events that frequently occur before failure conditions. Significance Analysis
helps to identify which events, and patterns of events, are the most significant
with respect to an error condition.
Visualization
The MDA includes tools that enable you to graphically drill into your machine
data and visualize previously hidden trends. In fact, all extracted data and the
sessionized records are available for processing by ad-hoc queries and visual-
ization in BigSheets. In addition to refining reports and queries, you can
explore your machine data using a powerful faceted search tool.
Faceted Search
After your machine data is in extracted form and has been indexed, you can
browse it using the Data Explorer graphical faceted search interface included
with BigInsights. This is a quick way to search machine data and expedite
