Database Reference
In-Depth Information
13/12/10 01:05:45 INFO mapred.JobClient: File Input Format Counters
13/12/10 01:05:45 INFO mapred.JobClient: Bytes Read=0
13/12/10 01:05:45 INFO mapred.JobClient: Map-Reduce Framework
13/12/10 01:05:45 INFO mapred.JobClient: Map input records=36153
13/12/10 01:05:45 INFO mapred.JobClient: Physical memory (bytes) snapshot=215248896
13/12/10 01:05:45 INFO mapred.JobClient: Spilled Records=0
13/12/10 01:05:45 INFO mapred.JobClient: CPU time spent (ms)=5452
13/12/10 01:05:45 INFO mapred.JobClient: Total committed heap usage (bytes)=514523136
13/12/10 01:05:45 INFO mapred.JobClient: Virtual memory (bytes) snapshot=653586432
13/12/10 01:05:45 INFO mapred.JobClient: Map output records=36153
13/12/10 01:05:45 INFO mapred.JobClient: SPLIT_RAW_BYTES=87
13/12/10 01:05:45 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 54.0554
seconds (0 bytes/sec)
13/12/10 01:05:45 INFO mapreduce.ImportJobBase: Retrieved 36153 records.
Windows PowerShell also provides cmdlets to execute Sqoop jobs. The following PowerShell script in Listing 6-10
exports the same
StockAnalysis
blob from WASB to a SQL Azure database called
ExportedData
.
Listing 6-10.
The Sqoop export PowerShell script
$subscriptionName= "Your_Subscription_Name"
$clusterName = "democluster"
$SqoopCommand = "export -connect
`"jdbc:sqlserver://<Server>.database.windows.net;username=debarchans@<Server>;
password=<Password>;database=sqoopdemo`" --table stock_analysis
--export-dir /user/hadoopuser/example/data/StockAnalysis
--input-fields-terminated-by `",`""
$sqoop = New-AzureHDInsightSqoopJobDefinition -Command $SqoopCommand
$SqoopJob = Start-AzureHDInsightJob -Subscription (Get-AzureSubscription
-Current).SubscriptionId -Cluster $clustername -JobDefinition $sqoop
Wait-AzureHDInsightJob -Subscription (Get-AzureSubscription
-Current).SubscriptionId -Job $SqoopJob -WaitTimeoutInSeconds 3600
Get-AzureHDInsightJobOutput -Cluster $clusterName -JobId $SqoopJob.JobId
-StandardError -Subscription $subscriptionName
Successful execution of the Sqoop export job shows output similar to Listing 6-11.
Listing 6-11.
The PowerShell Sqoop export output
StatusDirectory : ee84101c-98ac-4a2b-ae3d-49600eb5954b
ExitCode : 0
Name :
Query : export --connect
"jdbc:sqlserver://<Server>.database.windows.net;username=debarchans@<Server>;
password=<Password>;database=sqoopdemo" --table stock_analysis --export-dir
/user/hadoopuser/example/data/StockAnalysis --input-fields-terminated-by ","
State : Completed
SubmissionTime : 12/10/2013 1:36:36 AM
Cluster : democluster
PercentComplete : map 100% reduce 0%
JobId : job_201311240635_0205
