Databases Reference
In-Depth Information
of these discussions we can further define some of the features of next generation data
warehouses and data platforms:
•
Flexibility is paramount: The need is to have a “ask anything” data
warehouse.
•
It should be possible to store interim results: That is, you may
want to perform a query and use the output from that query
as a part of the input to another.
•
It should be easy to administer, cost effective, and offer a return
on investment in as short a timescale as is reasonable.
•
It should be efficient in terms of its resources: In particular, the
business analyst pursuing a line-of-thought inquiry should be
able to pursue any kind of a workload without being constrained
by big data characteristics.
•
Performance is also fundamental: While different queries will
obviously take different lengths of time, typical responses should
be in seconds, or minutes at most.
•
In modern-day enterprise data warehouses there is a growing
requirement to support a much larger number of users/queries
than was previously the case and, at the same time, a much
broader range of query types.
Data Warehouses and BI systems were built around the notion - data flows from
transactional systems possibly through staging areas to ODS to a centralized enterprise
data warehouse, the data from the EDW in turn then gets fed into the data marts of
various types, which then might feed personal databases. While it was often the case
that a single relational database would fulfill many of these data flow needs, this was not
always the case, especially where the data of interest is unstructured in nature.
In case of big data, the importance of data design and data flow is all the more
critical, as it's evident we'll have to deal with a mix of database technologies and
distributed architectures. In addition, we should also do careful considerations around
the value of data, as big data by very nature is considered to be full of noises whereas data
contained in the EDW is considered to be high quality and important to the organization.
Figure
4-6
illustrates a conceptual view of data flow architecture for big data scenarios.
Figure 4-6.
Big data flow
