Database Reference
In-Depth Information
• If your data is being created across many servers, do not create a central
collection point. This point will eventually create a bottleneck for your
loads. Instead, distribute the responsibility for publishing the data to
the cloud to the data producers.
• Consider your tooling. Most modern cloud storage tools support
uploading and downloading files in parallel.
• There is a tendency or desire to bundle multiple files into a single
compressed file before publishing to the cloud. This process will tax the
system responsible for packaging files and slow down your uploads.
Multiple small files will always upload faster than a single large or
monolithic file.
Exploring Big Data Storage Tools
AmazonS3andMicrosoftAzureBlobStorageprovidearobustRESTAPIfor
integrationwiththeirclouddataservices.Inaddition,anumberofwrappers
function as a layer of abstraction and provide easy access within code;
these are available in just about every development language. If you are a
developer (or even a closet developer), this may excite you and be exactly
what you are looking for. For most of you, however, you will prefer tools
that provide the same management and administration capabilities with no
coding required.
This section briefly introduces some of the more popular commercial tools
available and discusses the AzCopy project. The intent is not to endorse any
one tool over another, but instead to make you aware of the current state of
the marketplace.
Azure Management Studio
Developed by a company called Cerebata, Azure Management Studio
introduction
)
is a tool geared toward Microsoft Azure. It is a full-feature
product that not only allows for complete administration over your blob
storage but also includes other features such as pulling diagnostic and
miscellaneous analytical information associated with your account. It also
has a number of features built in to enable you to manage your other Azure
services. A demo version is available for evaluation, but you have to
purchase a license to use this product.
