Database Reference
In-Depth Information
Successfully added 'Microsoft.Hadoop.Hive 0.9.4951.25594' to HadoopClient.
Setting MRLib items CopyToOutputDirectory=true
Once the NuGet package has been added, add a reference to the .dll file in your code:
using Microsoft.Hadoop.Hive;
Once the references are added, you can develop the application code to construct and execute Hive queries
against your HDInsight cluster.
Creating the Hive Queries
The Hive .NET API exposes a few key methods to create and run Hive jobs. The steps are pretty similar to creating a
MapReduce job submission. Add a new
DoHiveOperations()
method in your
Program.cs file
. This method will
contain your Hive job submission code.
As with your MapReduce job submission code, the first step is to create your Hive job definition:
HiveJobCreateParameters hiveJobDefinition = new HiveJobCreateParameters()
{
JobName = "Show tables job",
StatusFolder = "/TableListFolder",
Query = "show tables;"
};
Next is the regular piece of code dealing with certificates and credentials to submit and run jobs in the cluster:
var store = new X509Store();
store.Open(OpenFlags.ReadOnly);
var cert = store.Certificates.Cast<X509Certificate2>().First(item =>
item.Thumbprint == Constants.thumbprint);
var creds = new JobSubmissionCertificateCredential(Constants.subscriptionId,
cert, Constants.clusterName);
Then create a job submission client object and submit the Hive job based on the definition:
var jobClient = JobSubmissionClientFactory.Connect(creds);
JobCreationResults jobResults = jobClient.CreateHiveJob(hiveJobDefinition);
Console.Write("Executing Hive Job.");
// Wait for the job to complete
WaitForJobCompletion(jobResults, jobClient);
Finally, you are ready to read the blob storage and display the output:
// Print the Hive job output
System.IO.Stream stream = jobClient.GetJobOutput(jobResults.JobId);
System.IO.StreamReader reader = new System.IO.StreamReader(stream);
Console.Write("Done..List of Tables are:\n");
Console.WriteLine(reader.ReadToEnd());
Listing 5-10 shows the complete
DoHiveOperations()
method. Note that it uses the same
WaitForJobCompletion()
method to wait and display progress while the job execution is in progress.
