Database Reference
In-Depth Information
On each node, make sure that the directories used in the configuration files exist:
[root@hc1nn conf]# mkdir -p /var/lib/hadoop-hdfs/cache/hdfs/dfs/name
Next, set the ownership of these directories:
[root@hc1nn conf]# chown -R hdfs:hdfs /var/lib/hadoop-hdfs/cache/hdfs/dfs/name
[root@hc1nn conf]# chmod 700 /var/lib/hadoop-hdfs/cache/hdfs/dfs/name
The preceding commands create the name directory, change the ownership to the hdfs user, and set its
permissions. Before starting the name node, though, you have to format the file system (as the hdfs user):
[root@hc1nn conf]# su - hdfs
-bash-4.1$ hdfs namenode -format
Now, you set up the file mapred-site.xml, setting the framework to be yarn :
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Next, you set up the file yarn-site.xml. There is a lot of configuration information here, including port addresses,
file system paths, and class path information. (For a full list of available configuration file options, check the Hadoop
site at hadoop.apache.org.)
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hc1nn:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hc1nn:8030</value>
</property>
 
Search WWH ::




Custom Search