Databases Reference
In-Depth Information
●
Metadata for data loading and processing will be implemented in file systems. A catalog of such
metadata will be versioned and stored using version control software.
●
The designated software stack such as Karmasphere or Predixion will manage semantic layers for
data output from the different data sets and their processes. The rules themselves will be stored in
a central repository for source code management.
●
All the preprocessing and processing of data will occur outside the data warehouse and other
relational data environments.
●
The hybrid data processing architecture will provide independent machine, software, and data
scalability across the entire ecosystem.
●
Any data integration between the new data outputs and the EDW will be done at a metric level
using metadata,
Armed with these directives the teams proceeded to analyze the available technology choices
and understand how they fit into the workload architecture they needed to design in order to cre-
ate a dynamic and scalable Big Data processing architecture. In summary this chapter provided you
with the fundamental insights into Big Data Processing Architectures and helped to learn the differ-
ences in processing structured data versus Big Data, and the case study has established the basic set
of requirements, processes, and potential architectures. The next chapter will be focussing on technol-
ogy choices for processing Big Data.
