Administering Hadoop - Hadoop: The Definitive Guide - page 392

Getting stack traces
Hadoop daemons expose a web page ( /stacks in the web UI) that produces a thread dump
for all running threads in the daemon's JVM. For example, you can get a thread dump for
a resource manager from http:// resource-manager-host :8088/stacks .
Metrics and JMX
The Hadoop daemons collect information about events and measurements that are collect-
ively known as metrics . For example, datanodes collect the following metrics (and many
more): the number of bytes written, the number of blocks replicated, and the number of
read requests from clients (both local and remote).
NOTE
The metrics system in Hadoop 2 and later is sometimes referred to as metrics2 to distinguish it from the
older (now deprecated) metrics system in earlier versions of Hadoop.
Metrics belong to a context ; “dfs,” “mapred,” “yarn,” and “rpc” are examples of different
contexts. Hadoop daemons usually collect metrics under several contexts. For example,
datanodes collect metrics for the “dfs” and “rpc” contexts.
HOW DO METRICS DIFFER FROM COUNTERS?
The main difference is their scope: metrics are collected by Hadoop daemons, whereas counters (see
Counters ) are collected for MapReduce tasks and aggregated for the whole job. They have different audi-
ences, too: broadly speaking, metrics are for administrators, and counters are for MapReduce users.
The way they are collected and aggregated is also different. Counters are a MapReduce feature, and the
MapReduce system ensures that counter values are propagated from the task JVMs where they are pro-
duced back to the application master, and finally back to the client running the MapReduce job. (Coun-
ters are propagated via RPC heartbeats; see Progress and Status Updates . ) Both the task process and the
application master perform aggregation.
The collection mechanism for metrics is decoupled from the component that receives the updates, and
there are various pluggable outputs, including local files, Ganglia, and JMX. The daemon collecting the
metrics performs aggregation on them before they are sent to the output.
All Hadoop metrics are published to JMX (Java Management Extensions), so you can use
standard JMX tools like JConsole (which comes with the JDK) to view them. For remote
monitoring, you must set the JMX system property com.sun.management.jmxre-
mote.port (and others for security) to allow access. To do this for the namenode, say,
you would set the following in hadoop-env.sh :