Database Reference
In-Depth Information
Now, the directories used by YARN, as shown in the configuration above, will need to be created and ownership
and group membership will have to be set to the YARN Linux user and group, as follows:
[root@hc1nn conf]# mkdir -p /var/log/hadoop-yarn/containers
[root@hc1nn conf]# mkdir -p /var/log/hadoop-yarn/apps
[root@hc1nn conf]# chown -R yarn:yarn /var/log/hadoop-yarn/containers
[root@hc1nn conf]# chown -R yarn:yarn /var/log/hadoop-yarn/apps
[root@hc1nn conf]# chmod 755 /var/log/hadoop-yarn/containers
[root@hc1nn conf]# chmod 755 /var/log/hadoop-yarn/apps
You add the following configuration to mapred-site.xml for the Job History server:
<property>
<name>mapreduce.jobhistory.address</name>
<value>hc1nn:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hc1nn:19888</value>
</property>
You also need to set up the staging directory in yarn-site.xml. If you don't specifically create one, YARN will create
a directory under /tmp. To create the directory, add the following xml to yarn-site.xml:
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/var/lib/hadoop-mapreduce/jobhistory/intermediate/donedir</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/var/lib/hadoop-mapreduce/jobhistory/donedir</value>
</property>
The directories needed for staging must be created on the file system. You set their ownership and group
membership to yarn , then set the permissions:
[root@hc1nn conf]# mkdir -p /var/lib/hadoop-mapreduce/jobhistory/intermediate/donedir
[root@hc1nn conf]# mkdir -p /var/lib/hadoop-mapreduce/jobhistory/donedir
[root@hc1nn conf]# chown -R yarn:yarn /var/lib/hadoop-mapreduce/jobhistory/intermediate/donedir
[root@hc1nn conf]# chown -R yarn:yarn /var/lib/hadoop-mapreduce/jobhistory/donedir
[root@hc1nn conf]# chmod 1777 /var/lib/hadoop-mapreduce/jobhistory/intermediate/donedir
[root@hc1nn conf]# chmod 750 /var/lib/hadoop-mapreduce/jobhistory/donedir
 
Search WWH ::




Custom Search