Database Reference
In-Depth Information
For Hadoop, we want to monitor these values for both the NameNode and
DataNodes. For the NameNodes, we want store and analyze the individual
values. For DataNodes, it is usually goodenough toreport on theaggregated
values of all DataNodes so that we can understand overall cluster utilization
and look for one of these particular resources causing a performance
bottleneck because of overutilization. In particular, you should monitor the
following:
• % CPU utilization and periodic (1-, 5-, 15-minute) CPU load averages
• Average disk data transfer rates (I/O bandwidth) and number of disk
I/O operations per second
• Average memory and swap space utilization
• Average data transfer rate and network latency
System Center Operations Manager for HDP
At the time of this writing, HDP for Windows does not include interfaces for
open source monitoring services such as Ganglia or Nagios. These services
are designed to consolidate information provided by Hadoop into a more
centralized and meaningful summary of services-related statistics in graphs
and alerts. On the other hand, Microsoft and Hortonworks have
collaborated to provide an Ambari System Center Operations Manager
(SCOM) solution. With this solution, SCOM can monitor availability,
capacity, and the health of the cluster and provide you valuable metrics,
graphs, and key performance indicators. In this section we'll look at the
overall capabilities of the Ambari MP for SCOM, walk through the
installation of the product, and finish by examining specific monitoring
scenarios.
TheAmbariprojectisaimedatmakingiteasiertomanageaHadoopcluster.
Amabari provides an interface and Rest APIs for provisioning, managing,
and monitoring a Hadoop cluster. Its interface is typically a Hadoop
management web UI that is powered by those Rest APIs. The Management
Pack for SCOM leverages those Rest APIs in order to provide the same
monitoring capabilities as the Ambari web UI, but in a familiar enterprise
solution like SCOM. Ambari SCOM will first automatically discover all
nodes within a Hadoop cluster, then proactively monitor availability,
capacity, and health, and finally provide visualizations within for
dashboards and trend analysis.
