Database Reference
In-Depth Information
the
fsck
output that lists all the files and blocks in the system, so you can compare it with
the output of running
fsck
after the upgrade.
It's also worth clearing out temporary files before doing the upgrade — both local tempor-
ary files and those in the MapReduce system directory on HDFS.
With these preliminaries out of the way, here is the high-level procedure for upgrading a
cluster when the filesystem layout needs to be migrated:
1. Ensure that any previous upgrade is finalized before proceeding with another up-
grade.
2. Shut down the YARN and MapReduce daemons.
3. Shut down HDFS, and back up the namenode directories.
4. Install the new version of Hadoop on the cluster and on clients.
5. Start HDFS with the
-upgrade
option.
6. Wait until the upgrade is complete.
7. Perform some sanity checks on HDFS.
8. Start the YARN and MapReduce daemons.
9. Roll back or finalize the upgrade (optional).
While running the upgrade procedure, it is a good idea to remove the Hadoop scripts from
your
PATH
environment variable. This forces you to be explicit about which version of
the scripts you are running. It can be convenient to define two environment variables for
the new installation directories; in the following instructions, we have defined
OLD_HADOOP_HOME
and
NEW_HADOOP_HOME
.
Start the upgrade
To perform the upgrade, run the following command (this is step 5 in the high-level up-
grade procedure):
%
$NEW_HADOOP_HOME/bin/start-dfs.sh -upgrade
This causes the namenode to upgrade its metadata, placing the previous version in a new
directory called
previous
under
dfs.namenode.name.dir
. Similarly, datanodes up-
grade their storage directories, preserving the old copy in a directory called
previous
.
Wait until the upgrade is complete
The upgrade process is not instantaneous, but you can check the progress of an upgrade
using
dfsadmin
(step 6; upgrade events also appear in the daemons' logfiles):
%
$NEW_HADOOP_HOME/bin/hdfs dfsadmin -upgradeProgress status
Upgrade for version -18 has been completed.
Upgrade is not finalized.
