Database Reference
In-Depth Information
This will copy all the
.csv
files under the
/debarchan/StockData
folder.
Once the source files are staged in your
WASB
, you need to define the Hive schema that will be a placeholder
for your Hive tables when you actually load data into it. Note that to run the PowerShell commands, you have to
download and install Windows Azure HDInsight PowerShell as described in Chapter 4. The HDInsight PowerShell
modules are integrated with Windows Azure Powershell version 0.7.2 and can be downloaded from
Execute the command in Listing 8-4 to create the Hive table.
Listing 8-4.
Creating the Hive table
stock_analysis
$subscriptionName = "YourSubscriptionName"
$storageAccountName = "democluster"
$containerName = "democlustercontainer"
$clustername = "democluster"
$querystring = "create external table stock_analysis
(stock_symbol string,stock_Date string,
stock_price_open double,stock_price_high double,
stock_price_low double,
stock_price_close double,
stock_volume int,
stock_price_adj_close double)
partitioned by (exchange string)
row format delimited
fields terminated by ','
LOCATION 'wasb://democlustercontainer@democluster.blob.core.windows.net/debarchan/StockData';"
■
You may need to wrap each of the commands in single line, depending on the powershell editor you use.
otherwise, you may encounter syntactical errors while running the script.
Note
$HiveJobDefinition = New-AzureHDInsightHiveJobDefinition -Query $querystring
$HiveJob = Start-AzureHDInsightJob -Subscription $subscriptionname
-Cluster $clustername -JobDefinition $HiveJobDefinition
$HiveJob | Wait-AzureHDInsightJob -Subscription $subscriptionname
-WaitTimeoutInSeconds 3600
Get-AzureHDInsightJobOutput -Cluster $clustername
-Subscription $subscriptionname
-JobId $HiveJob.JobId -StandardError
Once the job execution is complete, you should see output similar to the following:
StatusDirectory : 2b391c76-2d33-42c4-a116-d967eb11c115
ExitCode : 0
Name : Hive: create external table
Query : create external table stock_analysis
(stock_symbol string,stock_Date string,
stock_price_open double,stock_price_high double,
