Database Reference
In-Depth Information
v4.0_4.0.0.0__31bf3856ad364e35\PresentationCore.dll
[Auto -detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\UIAutomationProvider\
v4.0_4.0.0.0__31bf3856ad364e35\UIAutomationProvider.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\UIAutomationTypes\
v4.0_4.0.0.0__31bf3856ad364e35\UIAutomationTypes.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework.Aero2\
v4.0_4.0.0.0__31bf3856ad364e35\PresentationFramework.Aero2.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework-SystemXml\
v4.0_4.0.0.0__b77a5c561934e089\PresentationFramework-SystemXml.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework-SystemCore\
v4.0_4.0.0.0__b77a5c561934e089\PresentationFramework-SystemCore.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework-SystemData\
v4.0_4.0.0.0__b77a5c561934e089\PresentationFramework-SystemData.dll
[Auto-detected] D:\HadoopClient\HadoopClient\bin\Release\microsoft.hadoop.client.dll
[Auto-detected] D:\HadoopClient\HadoopClient\bin\Release\microsoft.hadoop.mapreduce.dll
[Auto-detected] D:\HadoopClient\HadoopClient\bin\Release\microsoft.hadoop.webclient.dll
[Auto-detected] D:\HadoopClient\HadoopClient\bin\Release\Newtonsoft.Json.dll
[Auto-detected] D:\HadoopClient\HadoopClient\bin\Release\HadoopClient.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework-SystemXmlLinq\
v4.0_4.0.0.0__b77a5c561934e089\PresentationFramework-SystemXmlLinq.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\UIAutomationClient\
v4.0_4.0.0.0__31bf3856ad364e35\UIAutomationClient.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationUI\
v4.0_4.0.0.0__31bf3856ad364e35\PresentationUI.dll
[Auto-detected] C:\windows\Microsoft.Net\assembly\GAC_MSIL\ReachFramework\
v4.0_4.0.0.0__31bf3856ad364e35\ReachFramework.dll
Job job_201309210954_0193 completed.
The MRRunner command can be put in a Windows batch file (.bat) or a command file (.cmd) and scheduled in
Windows Task Scheduler to execute it on a periodic basis. Of course, there are plenty of other ways as well to automate
MRRunner operations.
Summary
One of the major benefits of using the Azure HDInsight service is the elasticity it provides in terms of spinning up
clusters and running jobs exactly when they are required. The basic idea behind this is to avoid preserving idle
clusters just for storage. In HDInsight, the ultimate goal will be to present a script or a program that demonstrates
how you can provide a DLL and have the script bring a cluster online, run your job, and then remove the cluster, while
allowing you to specify the cluster name and the number of hosts needed to run the job. There are various ways you
can provision a new cluster, with the simplest of them being the Management portal. and it's easy-to-use, intuitive
graphical user interface. But as requirements become more and more complex and unpredictable along with project
budget limitations, automating and parameterizing cluster provisioning and job submissions become a necessity.
You can also provision cluster and configure it to connect to more than one Azure Blob storage or custom Hive and
Oozie metastores. This advanced feature allows you to separate lifetime of your data and metadata from the lifetime of
the cluster. There is a great sample script to provision an HDInsight cluster using custom configuration available at:
