Database Reference
In-Depth Information
Chapter 13
Big Data and the Cloud
What You Will Learn in This Chapter
• Exploring the Leading Cloud Providers: Amazon and Microsoft
• Using the Amazon Elastic Compute Cloud (EC2) and Microsoft
HDInsight Services as Cloud Processing Options
• Hosting and Storing Your Data in the Cloud with Amazon Simple Storage
Service (S3) and Microsoft Azure Blob Storage
Although many organizations will inevitably consider an on-premise solution
forbigdata,thebroadandever-expandingappealofthecloudmakesbigdata
approachable to those who would otherwise have neither the resources nor
the expertise. The focus of this chapter is threefold. First, we introduce the
cloud and look at two of the leading cloud providers: Amazon and Microsoft.
Next, we walk through setting up development or sandbox big data
environments to explore the Amazon Elastic Compute Cloud (EC2) service
and the Microsoft HDInsight service as cloud processing options.
We then explore two options—Amazon Simple Storage Service (S3) and
Microsoft Azure Blob Storage (ASV)—for hosting or storing your data in the
cloud. This discussion explains the considerations for cloud storage (pros/
cons), reviews best practices for storing your data in the cloud, and identifies
proven patterns and tools for loading and managing your data in the cloud.
Defining the Cloud
Oneofthemostoverused(andabused)contemporarytechnologytermsisthe
cloud
. You've probably seen or heard it used to describe just about everything
from distributed computing to the Internet. So, what is it, really? Well,
unfortunately, it is a little bit of everything, which is why we will limit our
context to that of applications, services, and (of course) data.
By changing the perspective in which we discuss the cloud, we can easily
clarify the definition. For our purposes, we will consider the cloud as the
hosted virtualization and abstraction of hardware that provides simple and
flexible solutions for capacity and scalability. This is sometimes referred to
