Database Reference
In-Depth Information
Monitoring
Monitoring is an important part of system administration. In this section, we look at the
monitoring facilities in Hadoop and how they can hook into external monitoring systems.
The purpose of monitoring is to detect when the cluster is not providing the expected level
of service. The master daemons are the most important to monitor: the namenodes (primary
and secondary) and the resource manager. Failure of datanodes and node managers is to be
expected, particularly on larger clusters, so you should provide extra capacity so that the
cluster can tolerate having a small percentage of dead nodes at any time.
In addition to the facilities described next, some administrators run test jobs on a periodic
basis as a test of the cluster's health.
Logging
All Hadoop daemons produce logfiles that can be very useful for finding out what is hap-
pening in the system.
System logfiles
explains how to configure these files.
Setting log levels
When debugging a problem, it is very convenient to be able to change the log level tempor-
arily for a particular component in the system.
Hadoop daemons have a web page for changing the log level for any log4j log name, which
can be found at
/logLevel
in the daemon's web UI. By convention, log names in Hadoop
correspond to the names of the classes doing the logging, although there are exceptions to
this rule, so you should consult the source code to find log names.
It's also possible to enable logging for all packages that start with a given prefix. For ex-
ample, to enable debug logging for all classes related to the resource manager, we would
visit the its web UI at
http://
resource-manager-host
:8088/logLevel
and set the log
name
org.apache.hadoop.yarn.server.resourcemanager
to level
DEBUG
.
The same thing can be achieved from the command line as follows:
%
hadoop daemonlog -setlevel
resource-manager-host
:8088 \
org.apache.hadoop.yarn.server.resourcemanager DEBUG
Log levels changed in this way are reset when the daemon restarts, which is usually what
you want. However, to make a persistent change to a log level, you can simply change the
log4j.properties
file in the configuration directory. In this case, the line to add is:
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager=DEBUG
