Databases Reference
In-Depth Information
FIGURE 10.4
Workload category.
Architecture
With the different data types clearly identified and laid out, the data characteristics, including the data
type, the associated metadata, the key data elements that can be identified as master data elements,
the complexities of the data, and the business users of the data from an ownership and stewardship
perspective, can be defined clearly.
Workload
The biggest need for processing Big Data is workload management, as discussed in earlier chapters.
The data architecture and classification allow us to assign the appropriate infrastructure that can exe-
cute the workload demands of the categories of the data.
There are four broad categories of workload based on volume of data and the associated laten-
cies that data can be assigned to (
Figure 10.4
). Depending on the type of category the data can then
be assigned to physical infrastructure layers for processing. This approach to workload management
creates a dynamic scalability requirement for all parts of the Data Warehouse, which can be designed
by efficiently harnessing the current and new infrastructure options. The key point to remember at this
juncture is the processing logic needs to be flexible to be implemented across the different physical
infrastructure components since the same data might be classified into different workloads depending
on the urgency of processing.
The workload architecture will further identify the conditions of mixed workload management
where the data from one category of workload will be added to processing along with another cat-
egory of workload.
For example, processing high-volume, low-latency data with low-volume, high-latency data cre-
ates a diversified stress on the data processing environment, where you normally would have pro-
cessed one kind of data and its workload. Add to this complexity the user query and data loading
happening at the same time or in relatively short intervals, and now the situation can get out of hand
in a quick succession and impact the overall performance. If the same infrastructure is processing Big
Data and traditional data together with all of these complexities, the problem just compounds itself.
