Database Reference
In-Depth Information
sec per (physical) core of data. Therefore, the maximum core count places
a cap on the storage capacity a system can meaningfully address without
adversely affecting query performance.
Moving forward, a 120TB cap might prove to be too small. However, the
truth is that 120TB cap is already too small. The rise of Hadoop is testament
to that. Data volumes are growing exponentially and are therefore driving
storage requirements that need to be measured by a multiplier, not a
percentage.Thatmakesanorderofmagnitudeofdifferencetothesizeofthe
warehouse and, as architects, is something we need to design for.
So, where is this data coming from? Well, clearly, we can look to things such
as social media, machine data, sensor data, and web analytics as examples
of new data sources driving this change. However, there are other reasons
too.
Historically, companies (especially in Europe, the Middle East, and Africa
[EMEA]) have been good at archiving data and taking it offline (storing it
on tape or some other media). Perhaps they've only been keeping a couple
of years of data online. This is changing. Companies are being asked to keep
more data online, either to feed analytical engines for training new models
or for regulatory reasons. Businesses want to query over all their data and
are much less tolerant of delays in response while data is retrieved from
warm or cold storage.
In addition, companies are starting to exchange and sell data as products to
other companies. Retailers have been buying data for years to aid analytics.
Consider weather data as a classic example. How will the weather affect
sales of ice cream? That seems easy enough. Depending on the granularity
of the data requested, that may make a modest bump in the size of a
warehouse.
Nowconsideranothersource:Thinkaboutthelocationdatathatyourphone
offers up to every app provider. Where have you been? What routes do you
usuallytravel?Whichshopsdoyouactuallyvisit?Allmanner ofinformation
is suddenly available, providing a much richer data set for analysts to
review. However, not only is this data a rich source of information, it is
also potentially truly massive. Imagine tacking every telco's warehouse onto
yours. Will Smith was certainly onto something when he portentously
starred in
Enemy of the State
.
