Database Reference
In-Depth Information
Table 12-1.
(
continued
)
Status
What it means
Windows Azure VM
Configuration
The HDInsight Deployment Service makes calls to Azure to initiate the provisioning
of virtual machines (VMs) for the Head Node, Worker Nodes, and Gateway Node(s).
The gateway acts as the security boundary between the cluster and the outside world.
All traffic coming into the cluster goes through the gateway for authentication and
authorization. The gateway can be thought of as a proxy that performs the necessary
operations and forwards the request to the appropriate cluster components.
So if you try to connect through the templeton or hive from, say, Excel, the call enters the
gateway and then is proxied through to the rest of the components.
HDInsight Configuration
On startup, each node runs custom actions that download and install the appropriate
components. These actions are coordinated by the individual node's local Deployment
Agent. Installations of the Java Runtime, Hortonworks Data Platform, Microsoft HDInsight
on Azure, and component bundles like Hive, Pig, Hcatalog, Sqoop and Oozie are run.
Running
The cluster is ready for use.
A few scenarios apart from the ones shown in the preceding table can lead to failure during the
cluster-provisioning process:
•
Race condition exists on cluster creation.
An operation to create the hidden Cloud Service
object was not synchronous. A subsequent call to retrieve the Cloud Service to use in the next
step failed.
•
VM cores are limited by subscription.
Attempts to create a cluster using cores past the
subscription limit failed.
•
Datacenter capacity is limited.
Because HDInsight clusters can use a large number of cores.
Cluster creation failures can occur when the datacenter is near capacity.
•
Certain operations have must-succeed logging attached to them.
If the underlying logging
infrastructure (Windows Azure Tables) is not available or times out, the cluster creation
effort may fail.
Installer Logs
The Windows Azure HDInsight Service has a mechanism to log its cluster-deployment operations. Log files are placed
in the
C:\HDInsightLogs\
directory in the name node and data nodes. They contain two types of log files:
•
AzureInstallHelper.log
•
DeploymentAgent.log
These files give you information about several key aspects of the deployment process. Basically, after the VMs
are provisioned, a deployment service runs for HDInsight that unpacks and installs Hadoop and its supporting
projects with the necessary Windows services on the name node and data nodes. For example, if a node re-imaging
has taken place, there will be re-imaging status entries at the very beginning of the
DeploymentAgent.log
file,
as shown in Listing 12-1.
