Database Reference
In-Depth Information
SSH settings
The control scripts allow you to run commands on (remote) worker nodes from the master
node using SSH. It can be useful to customize the SSH settings, for various reasons. For
example, you may want to reduce the connection timeout (using the
ConnectTimeout
option) so the control scripts don't hang around waiting to see whether a dead node is go-
ing to respond. Obviously, this can be taken too far. If the timeout is too low, then busy
nodes will be skipped, which is bad.
Another useful SSH setting is
StrictHostKeyChecking
, which can be set to
no
to
automatically add new host keys to the known hosts files. The default,
ask
, prompts the
user to confirm that the key fingerprint has been verified, which is not a suitable setting in
a large cluster environment.
[
72
]
To pass extra options to SSH, define the
HADOOP_SSH_OPTS
environment variable in
hadoop-env.sh
. See the
ssh
and
ssh_config
manual pages for more SSH settings.
Important Hadoop Daemon Properties
Hadoop has a bewildering number of configuration properties. In this section, we address
the ones that you need to define (or at least understand why the default is appropriate) for
any real-world working cluster. These properties are set in the Hadoop site files:
core-
site.xml
,
hdfs-site.xml
, and
yarn-site.xml
. Typical instances of these files are shown in
figuration files in
The Configuration API
.
To find the actual configuration of a running daemon, visit the
/conf
page on its web serv-
er. For example,
http://
resource-manager-host
:8088/conf
shows the configuration
that the resource manager is running with. This page shows the combined site and default
configuration files that the daemon is running with, and also shows which file each prop-
erty was picked up from.
Example 10-1. A typical core-site.xml configuration file
<?xml version="1.0"?>
<!-- core-site.xml -->
<configuration>
<property>
<name>
fs.defaultFS
</name>
<value>
hdfs://namenode/
</value>
</property>
</configuration>
Example 10-2. A typical hdfs-site.xml configuration file
