Databases Reference
In-Depth Information
Chapter 6
Data Modeling Approaches
for Big Data and Analytics
Solutions
One common theme you will hear again and again concerning big data
solutions: there is no schema to model! Does this mean we do not need to
do any data modeling activities while constructing a big data solution?
Data integration, in effect is theacquisition of data from diverse source systems
(like operational applications for ERP, CRM, supply chain, where most enterprise data
originates and a host of external sources of data like social networks, external third party
data sources, etc.) through multiple transformations of the data to get it ready for loading
into target systems (like data warehouses, customer data hubs, and product catalogs).
Heterogeneity is the norm for both data sources and targets, since there are various types
of applications, databases, file types, and so on. All these have different data models,
so the data must be transformed in the middle of the process, and the transformations
themselves vary widely. Then there are the interfaces that connect these pieces, which are
equally diverse. And the data doesn't flow uninterrupted or in a straight line, so you need
data staging areas. Simply put, that's a lot of complex and diverse activities that you must
perform to organize data to make it useful.
Eventually the data integration processes and approaches influence the data model
development as well. Let's first understand the data integration patterns.
Understanding Data Integration Patterns
Data integration approaches can become highly complex especially when you are dealing with
big data types. Below is an attempt to outline the complexities of data integration processes.
•
Level 0:
Simple point to point data integration with little or no
transformation. This just means information is flowing from one
system to another.
•
Level 1:
Simple data integration processes, transforming one
schema to another, without applying any data manipulation
functions like “if,” “then,” “else,” etc.
