Database Reference
In-Depth Information
dfs.datanode.dns.interface
, which is set to
default
to use the default net-
work interface. You can set this explicitly to report the address of a particular interface
(
eth0
, for example).
Other Hadoop Properties
This section discusses some other properties that you might consider setting.
Cluster membership
To aid in the addition and removal of nodes in the future, you can specify a file containing
a list of authorized machines that may join the cluster as datanodes or node managers. The
file is specified using the
dfs.hosts
and
yarn.resourcemanager.nodes.include-path
properties (for datanodes and
node managers, respectively), and the corresponding
dfs.hosts.exclude
and
yarn.resourcemanager.nodes.exclude-path
properties specify the files
used for decommissioning. See
Commissioning and Decommissioning Nodes
for further
discussion.
Buffer size
Hadoop uses a buffer size of 4 KB (4,096 bytes) for its I/O operations. This is a conservat-
ive setting, and with modern hardware and operating systems, you will likely see perform-
ance benefits by increasing it; 128 KB (131,072 bytes) is a common choice. Set the value
in bytes using the
io.file.buffer.size
property in
core-site.xml
.
HDFS block size
The HDFS block size is 128 MB by default, but many clusters use more (e.g., 256 MB,
which is 268,435,456 bytes) to ease memory pressure on the namenode and to give map-
pers more data to work on. Use the
dfs.blocksize
property in
hdfs-site.xml
to specify
the size in bytes.
Reserved storage space
By default, datanodes will try to use all of the space available in their storage directories.
If you want to reserve some space on the storage volumes for non-HDFS use, you can set
dfs.datanode.du.reserved
to the amount, in bytes, of space to reserve.
