Database Reference
In-Depth Information
that these properties refer to the same file. The exclude file lists the nodes that are not per-
mitted to connect to the cluster.
The rules for whether a node manager may connect to the resource manager are simple: a
node manager may connect only if it appears in the include file and does not appear in the
exclude file. An unspecified or empty include file is taken to mean that all nodes are in the
include file.
For HDFS, the rules are slightly different. If a datanode appears in both the include and
the exclude file, then it may connect, but only to be decommissioned.
Table 11-3
summar-
izes the different combinations for datanodes. As for node managers, an unspecified or
empty include file means all nodes are included.
Table 11-3. HDFS include and exclude file precedence
Node appears in include
file
Node appears in exclude
file
Interpretation
No
No
Node may not connect.
No
Yes
Node may not connect.
Yes
No
Node may connect.
Yes
Yes
Node may connect and will be decommis-
sioned.
To remove nodes from the cluster:
1. Add the network addresses of the nodes to be decommissioned to the exclude file.
Do not update the include file at this point.
2. Update the namenode with the new set of permitted datanodes, using this com-
mand:
%
hdfs dfsadmin -refreshNodes
3. Update the resource manager with the new set of permitted node managers using:
%
yarn rmadmin -refreshNodes
4. Go to the web UI and check whether the admin state has changed to “Decommis-
sion In Progress” for the datanodes being decommissioned. They will start copy-
ing their blocks to other datanodes in the cluster.
5. When all the datanodes report their state as “Decommissioned,” all the blocks
have been replicated. Shut down the decommissioned nodes.
6. Remove the nodes from the include file, and run:
