Databases Reference
In-Depth Information
The years 2000 to 2010 have been a defining moment in the history of data, emergence of search
engines (Google, Yahoo), personalization of music (iPod), tablet computing (iPad), bigger mobile
solutions (smartphones, 3G networks, mobile broadband, Wi-Fi), and emergence of social media
(driven by Facebook, MySpace, Twitter, and Blogger). All these entities have contributed to the con-
sumerization of data, from data creation, acquisition, and consumption perspectives.
The business models and opportunities that came with the large-scale growth of data drove the
need to create powerful metrics to tap from the knowledge of the crowd that was driving them, and in
return offer personalized services to address the need of the moment. This challenge was not limited
to technology companies; large multinational organizations like P&G and Unilever wanted solutions
that could address data processing, and additionally wanted to implement the output from large-scale
data processing into their existing analytics platform.
Google, Yahoo, Facebook, and several other companies invested in technology solutions for data
management, allowing us to consume large volumes of data in a short amount of time across many
formats with varying degrees of complexity to create a powerful decision support platform. These
technologies and their implementation are discussed in detail in later chapters in this topic.
Defining Big Data
Big Data can be defined as volumes of data available in varying degrees of complexity, generated at
different velocities and varying degrees of ambiguity, that cannot be processed using traditional tech-
nologies, processing methods, algorithms, or any commercial off-the-shelf solutions.
Data defined as Big Data includes machine-generated data from sensor networks, nuclear plants,
X-ray and scanning devices, and airplane engines, and consumer-driven data from social media. Big
Data producers that exist within organizations include legal, sales, marketing, procurement, finance,
and human resources departments.
Why Big Data and why now?
These are the two most popular questions that are crossing the minds of any computing professional:
Why Big Data? Why now? The promise of Big Data is the ability to access large volumes of data
that can be useful in gaining critical insights from processing repeated or unique patterns of data or
behaviors. This learning process can be executed as a machine-managed process with minimal human
intervention, making the analysis simpler and error-free. The answer to the second question—Why
now?—is the availability of commodity infrastructure combined with new data processing frame-
works and platforms like Hadoop and NoSQL, resulting in significantly lower costs and higher scal-
ability than traditional data management platforms. The scalability and processing architecture of the
new platforms were limitations of traditional data processing technologies, though the algorithms and
methods existed.
The key thing to understand here is the data part of Big Data was always present and used in a
manual fashion, with a lot of human processing and analytic refinement, eventually being used in a
decision-making process. What has changed and created the buzz with Big Data is the automated data
processing capability that is extremely fast, scalable, and has flexible processing.
