Database Reference
In-Depth Information
Think about it for a moment. The opportunity cost clock on your data
starts ticking the moment the data hits the wire. As organizations, we're tak-
ing far too long to spot trends or pick up valuable insights. It doesn't matter
what industry you're in; being able to more swiftly understand and respond
to data signals puts you in a position of power. Whether you're trying to
understand the health of a traffic system, the health of a patient, or the health
of a loan portfolio, reacting faster gives you an advantage. Velocity is per-
haps one of the most overlooked areas in the Big Data craze, and one in
which we believe that IBM is unequalled in the capabilities and sophistica-
tion that it provides.
In the Big Data craze that has taken the marketplace by storm, everyone
is fixated on at-rest analytics, using optimized engines such the Netezza
technology behind the IBM PureData System for Analytics or Hadoop to
perform analysis that was never before possible, at least not at such a large
scale. Although this is vitally important, we must nevertheless ask: “How
do you analyze data in motion?” This capability has the potential to provide
businesses with the highest level of differentiation, yet it seems to be somewhat
overlooked. The IBM InfoSphere Streams (Streams) part of the IBM Big Data
platform provides a real-time streaming data analytics engine. Streams is a
platform that provides fast, flexible, and scalable processing of continuous
streams of time-sequenced data packets. We'll delve into the details and
capabilities of Streams in Part III, “Analytics for Big Data in Motion.”
You might be thinking that velocity can be handled by Complex Event
Processing (CEP) systems, and although they might seem applicable on the
surface, in the Big Data world, they fall very short. Stream processing enables
advanced analysis across diverse data types with very high messaging data
rates and very low latency (µs to s). For example, one financial services sector
(FSS) client analyzes and correlates over five million market messages/
second to execute algorithmic option trades with an average latency of 30
microseconds. Another client analyzes over 500,000 Internet protocol detail
records (IPDRs) per second, more than 6 billion IPDRs per day, on more than
4PB of data per year, to understand the trending and current-state health of their
network. Consider an enterprise network security problem. In this domain,
threats come in microseconds so you need technology that can respond and
keep pace. However you also need something that can capture lots of data
