Setting Up a Hadoop Cluster - Hadoop: The Definitive Guide - page 343

Hadoop Configuration

There are a handful of files for controlling the configuration of a Hadoop installation; the
most important ones are listed in Table 10-1 .

Table 10-1. Hadoop configuration files

Filename Format Description

hadoop-env.sh Bash script Environment variables that are used in the scripts to run Hadoop

mapred-env.sh Bash script Environment variables that are used in the scripts to run MapReduce

(overrides variables set in hadoop-env.sh )

yarn-env.sh Bash script Environment variables that are used in the scripts to run YARN (over-

rides variables set in hadoop-env.sh )

core-site.xml Hadoop config-

uration XML

Configuration settings for Hadoop Core, such as I/O settings that are
common to HDFS, MapReduce, and YARN

hdfs-site.xml Hadoop config-

uration XML

Configuration settings for HDFS daemons: the namenode, the second-
ary namenode, and the datanodes

mapred-site.xml Hadoop config-
uration XML

Configuration settings for MapReduce daemons: the job history server

yarn-site.xml Hadoop config-

uration XML

Configuration settings for YARN daemons: the resource manager, the
web app proxy server, and the node managers

slaves Plain text A list of machines (one per line) that each run a datanode and a node

manager

hadoop-metrics2
.properties

Java properties Properties for controlling how metrics are published in Hadoop (see
Metrics and JMX )

log4j.properties Java properties Properties for system logfiles, the namenode audit log, and the task log
for the task JVM process ( Hadoop Logs )

hadoop-
policy.xml

Hadoop config-
uration XML

Configuration settings for access control lists when running Hadoop in
secure mode

These files are all found in the etc/hadoop directory of the Hadoop distribution. The con-
figuration directory can be relocated to another part of the filesystem (outside the Hadoop
installation, which makes upgrades marginally easier) as long as daemons are started with
the --config option (or, equivalently, with the HADOOP_CONF_DIR environment vari-
able set) specifying the location of this directory on the local filesystem.