Database Reference
In-Depth Information
$inputPath = "wasb:///example/data/gutenberg/davinci.txt"
$outputPath = "wasb:///example/data/WordCountOutput"
$jarFile = "wasb:///example/jars/hadoop-examples.jar"
$class = "wordcount"
$passwd = ConvertTo-SecureString "Your_Password" -AsPlainText -Force
$myCreds = New-Object System.Management.Automation.PSCredential ("admin", $secpasswd)
# Define the word count MapReduce job
$mapReduceJobDefinition = New-AzureHDInsightMapReduceJobDefinition -JarFile $jarFile
-ClassName $class -Arguments $inputPath, $outputPath
# Submit the MapReduce job
Select-AzureSubscription $subscription
$wordCountJob = Start-AzureHDInsightJob -Cluster $cluster -JobDefinition
$mapReduceJobDefinition -Credential $myCreds
# Wait for the job to complete
Wait-AzureHDInsightJob -Job $wordCountJob -WaitTimeoutInSeconds 3600 -Credential $myCreds
# Get the job standard error output
Get-AzureHDInsightJobOutput -Cluster $cluster -JobId $wordCountJob.JobId -StandardError
-Subscription $subscription
# Get the blob content
Get-AzureStorageBlobContent -Container $Container -Blob example/data/WordCountOutputPS/part-r-00000
-Context $storageContext -Force
# List the content of the output file
cat ./example/data/WordCountOutputPS/part-r-00000 | findstr "human"
Executing The Job
You can execute the script directly from PowerShell ISE or use the Windows Azure PowerShell command prompt.
Save the script file as
SubmitJob.ps1
in a location of your choice, and execute it from the PowerShell prompt. You
should see an output similar to the following once the script completes successfully:
PS C:\> C:\SubmitJob.ps1
StatusDirectory : 0fac8406-891d-41ff-af74-eaac21386fd3
ExitCode : 0
Name : wordcount
Query :
State : Completed
SubmissionTime : 12/9/2013 7:47:05 PM
Cluster : democluster
PercentComplete : map 100% reduce 100%
JobId : job_201311240635_0192
13/12/09 19:47:19 INFO input.FileInputFormat: Total input paths to process : 1
13/12/09 19:47:19 WARN snappy.LoadSnappy: Snappy native library is available
13/12/09 19:47:19 INFO util.NativeCodeLoader: Loaded the native-hadoop library
