Database Reference
In-Depth Information
out as well. You might choose to use the Ambari project for this purpose.
Alternatively, youcouldstaywithin theMicrosoft ecosystem and useSystem
Center to drive your monitoring. An Ambari SCOM Management Pack has
been created so that you can monitor Hadoop using the same infrastructure
as the rest of your enterprise. Either way, you need to ensure that you have
integrated the new infrastructure into your environment and make sure that
all the feeds of data can access the new cluster.
So, why follow this model? Speaking personally, it often comes down to
flexibility, proximity, security, and functionality. However, proximity is one
factor to emphasize here. If the data is born on premise, and in significant
volume, it may make greater sense to retain the data in an on-premise
environment. Pushing large volumes of data to the cloud might be
impractical and might introduce too much latency prior to the analysis.
That said, this issue may be tempered once the initial data load has been
accomplished. Subsequent loads will contain only delta changes which
would hopefully reduce the burden of the latency.
Remember, though, the second part of proximity, which pertains to
integration with other data environments. Business users don't actually care
where or what store their data is; they just want to query (and that might be
a query across environments).
PDW Integration
Querying across environments was one of the three key design goals for
project Polybase, one of the stand-out features in SQL Server Parallel Data
Warehouse (PDW) 2012. Polybase provides the glue between Hadoop and
the data warehouse for the parallel import and export of data. However, it
also enables a unified heterogeneous querying platform for end users. In
other words, I can write the following and it just works:
SELECT COUNT(*)
, SUM(s.Value) AS Total_Sales
, p.Category_name
, d.CalendarMonth_name
FROM dbo.hdfs_Sales s
JOIN dbo.pdw_Product p ON
s.Product_key = p.Product_key
JOIN dbo.pdw_Date d ON
s.Date_ = p.Date_key
