Database Reference
In-Depth Information
To demonstrate, I have borrowed this file from the V1 configuration:
<allocations>
<pool name="high_pool">
<minMaps>10</minMaps>
<minReduces>10</minReduces>
<maxMaps>50</maxMaps>
<maxReduces>50</maxReduces>
<maxRunningJobs>1000</maxRunningJobs>
<weight>3</weight>
</pool>
<pool name="low_pool">
<minMaps>10</minMaps>
<minReduces>10</minReduces>
<maxMaps>50</maxMaps>
<maxReduces>50</maxReduces>
<maxRunningJobs>1000</maxRunningJobs>
<weight>1</weight>
</pool>
<pool name="default">
<minMaps>10</minMaps>
<minReduces>10</minReduces>
<maxMaps>50</maxMaps>
<maxReduces>50</maxReduces>
<maxRunningJobs>1000</maxRunningJobs>
<weight>1</weight>
</pool>
</allocations>
You now use the yarn rmadmin command to refresh the YARN scheduler queue configuration. This will cause
YARN (as for V2 Capacity) to reread its configuration files and so pick up the changes that you have made:
[hadoop@hc1nn conf]$ yarn rmadmin -refreshQueues
To demonstrate its use, I create a Map Reduce job and specify a queue value of high_queue so that I can ensure
that the submitted job will be processed by the YARN Fair scheduler and will be placed in the right queue.
The job I use specifies the queue name using a -D option:
hadoop \
jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapred.job.queue.name=high_pool \
/usr/hadoop/edgar \
/usr/hadoop/edgar-results1
Figure 5-4 shows the job status in the scheduler user interface. The queue legend now displays Fair Share, and
the job details appear in the high_pool queue, which is marked in green to indicate it's being used.
 
Search WWH ::




Custom Search