Database Reference
In-Depth Information
Chapter 1
Introduction to Streaming Data
It seems like the world moves at a faster pace every day. People and places
become more connected, and people and organizations try to react at an
ever-increasing pace. Reaching the limits of a human's ability to respond,
tools are built to process the vast amounts of data available to decision
makers, analyze it, present it, and, in some cases, respond to events as they
happen.
The collection and processing of this data has a number of application areas,
some of which are discussed in the next section. These applications, which
are discussed later in this chapter, require an infrastructure and method of
analysis specific to streaming data. Fortunately, like batch processing before
it, the state of the art of streaming infrastructure is focused on using
commodity hardware and software to build its systems rather than the
specialized systems required for real-time analysis prior to the Internet era.
This, combined with flexible cloud-based environment, puts the
implementation of a real-time system within the reach of nearly any
organization. These commodity systems allow organizations to analyze their
data in real time and scale that infrastructure to meet future needs as the
organization grows and changes over time.
The goal of this topic is to allow a fairly broad range of potential users and
implementers in an organization to gain comfort with the complete stack of
applications. When real-time projects reach a certain point, they should be
agile and adaptable systems that can be easily modified, which requires that
the users have a fair understanding of the stack as a whole in addition to
their own areas of focus. “Real time” applies as much to the development
of new analyses as it does to the data itself. Any number of well-meaning
projects have failed because they took so long to implement that the people
who requested the project have either moved on to other things or simply
forgotten why they wanted the data in the first place. By making the projects
agile and incremental, this can be avoided as much as possible.
This chapter is divided into sections that cover three topics. The first section,
“Sources of Streaming Data,” is some of the common sources and
applications of streaming data. They are arranged more or less
