Database Reference
In-Depth Information
In an ideal world, we would never have to spend so much time unpacking and solving
data challenges. An ideal data store would have all the features we need to build our
applications. It would have the availability of a key-value or document-oriented data-
base, but would provide a relational model of storing data for the best possible consis-
tency. The database would be hosted as a service in the cloud so that no infrastructure
would have to be purchased or managed. This system would be infinitely scalable and
would work the same way if the amount of data under management consisted of one
megabyte or 100 terabytes. In essence, this database solution would be the magical,
infinitely scalable, always available database in the sky.
As of this publication, there is currently no such magic database in the sky—
although there are many efforts to commercialize cutting-edge database technology
that combine many of the different data software paradigms we mentioned earlier in
the chapter.
Some companies have attempted to create a similar product by providing each of
the various steps in the data pipeline—from highly available data collection to trans-
formation to storage caching and analysis—behind a unified interface that hides some
of these complexities.
Solving large-scale data challenges ultimately boils down to building a scalable strategy
for tackling well-defined, practical use cases. The best solutions combine technologies
designed to tackle specific needs for each step in a data processing pipeline. Provid-
ing high availability along with the caching of large amounts of data as well as high-
performance analysis tools may require coordination of several sets of technologies.
Along with this, more complex pipelines may require data-transformation techniques
and the use of specific formats designed for efficient sharing and interoperability.
The key to making the best data-strategy decisions is to keep our core data prin-
ciples in mind. Always understand your business needs and use cases before evaluating
technology. When necessary, make sure that you have a plan to scale your data solu-
tion—either by deciding on a database that can handle massive growth of data or by
having a plan for interoperability when the need for new software comes along. Make
sure that you can retrieve and export data. Think about strategies for sharing data,
whether internally or externally. Avoid the need to buy and manage new hardware.
And above all else, always keep the questions you are trying to answer in mind before
embarking on a software development project.
Now that we've established some of the ground rules for playing the game in the
Era of the Big Data Trade-Off, let's take a look at some winning game plans.
Search WWH ::
Custom Search