Database Reference
In-Depth Information
The IBM Accelerator for
Machine Data Analytics
Today's enterprises depend heavily on the uptime of their IT infrastructure.
One of our clients makes money through a stock ticker-like application that
conveys noteworthy news items—if it isn't running, they aren't making
money. Another online retailer estimates that 17 kinds of logs are generated
in the process of an online order. Web logs, application server logs, Hibernate
logs, database logs, driver logs, and more, all contain discrete views into the
transaction as a whole. These clients tell us they can't afford downtime, so
they put a lot of investment and smarts into areas of concern. We think for as
much thought and planning that goes into keeping systems highly available
and recoverable from disaster, we still take too much for granted when it
comes to the highly interconnected nature of our IT environments, where
everyone's devices are networked, and a multitude of systems pump infor-
mation through this maze of wires. Of course, when something goes wrong,
our dependencies on these systems become painfully obvious. The activities
of entire departments can be brought to a standstill without this lifeblood of
connectivity. As such, when IT outages happen, there's an extreme sense of
urgency around restoring connectivity. Here's the thing: if uptime is so criti-
cal, why are so few of us making a corpus of log information and finding
trendline correlations between log events that seem harmless in isolation, but
taken in context with other IT activities prove to be the root cause for a down-
stream outage? Big Data technologies give businesses an unprecedented
opportunity to create insight into this myriad of log files that yields hints
and clues on not just what went wrong in the past, but how to prevent
things going sideways in the future.
As we alluded to in the previous paragraph, two main factors make IT out-
ages challenging today: the high degree of interconnectedness and growing
interdependencies between IT systems, and the high volume of usage against
these systems. The key to diagnosing the root causes of failures involves sys-
tem administrators combing through IT logs (also known as machine data)
from their various servers and pinpointing the origin of the chain of events
that led to an outage. And it's the nature of these logs that pose the biggest
challenge. All these different systems store their logs in different locations, use
different file formats, and use different presentation styles for elements such
