Databases Reference
In-Depth Information
more processing capabilities, access to wider memory, and have accelerated architecture evolution
within the software layers.
●
Memory
. While the storage of data to disk for offline processing proved the need for storage
evolution and data management, equally important was the need to store data in perishable
formats in memory for compute and processing. Memory has become cheaper and faster, and with
the evolution of processor capability, the amount of memory that can be allocated to a system,
then to a process within a system, has changed significantly.
●
Software.
Another core data processing component is the software used to develop the programs
to transform and process the data. Software across different layers from operating systems to
programming languages has evolved generationally and even leapfrogged hardware evolution
in some cases. In its lowest form the software translates sequenced instruction sets into machine
language that is used to process data with the infrastructure layers of CPU + memory + storage.
Programming languages that have evolved over time have harvested the infrastructure evolution
to improve the speed and scalability of the system. Operating systems like Linux have opened the
doors of innovation to enterprises to develop additional capabilities to the base software platform
for leveraging the entire infrastructure and processing architecture improvements.
Speed or throughput
The biggest continuing challenge is the speed or throughput of data processing. Speed is a combina-
tion of various architecture layers: hardware, software, networking, and storage. Each layer has its
own limitations and, in a combination, these limitations have challenged the overall throughput of
data processing.
Data processing challenges continue to exist in the infrastructure architecture layers as an ecosys-
tem, though the underlying software, processor, memory, storage, and network components have all
evolved independently. In the world of database processing and data management this is a significant
problem both from a value and a financial perspective. In the next section we discuss the architectures
that were initially developed as shared-everything architecture and the problems that were solved as
transaction processing on these platforms, and the newer evolution to shared-nothing architecture
that has given us the appliance platform, which is providing the unlimited scalability that was always
lacking in the world of data and its management.
Shared-everything and shared-nothing architectures
Data processing is an intense workload that can either scale dramatically or severely underperform
and crash. The key to both the scenarios stems from the underlying infrastructure architecture, based
on which a particular data architecture performance can be predicted. Two popular data processing
infrastructure architectures that are regarded as industry standard are shared-everything and shared-
nothing architectures.
Application software such as CRM, ERP, SCM, and transaction processing require software that
can drive performance. Web applications require an architecture that is scalable and flexible. Data
warehousing requires an infrastructure platform that is robust and scalable.
Based on the nature of the data and the type of processing, shared-everything architectures are suited
for applications, while shared-nothing architecture lends itself to data warehouse and web applications.
