Database Reference
In-Depth Information
Job submission in YARN
The following are the sequence of steps involved when a job is submitted to a YARN
cluster:
1. When a job is submitted to the cluster, the client first receives an application ID
from the ResourceManager.
2. Next, the client copies the job resources to a location in the HDFS.
3. The ResourceManager then starts the first container under the NodeManager's
management to bring up the ApplicationMaster. For example, if a MapReduce job
is submitted, the ResourceManager will bring up the MapReduce Applica-
tionMaster.
4. The ApplicationMaster, based on the job to be executed, requests resources from
the ResourceManager.
5. Once the ResourceManager schedules a container with the requested resource, the
ApplicationMaster contacts the NodeManager to start the container and execute the
task. In case of a MapReduce job, that task would be a map or reduce task.
6. The client checks with the ApplicationMaster for status updates on the submitted
job.
The following diagram shows the interactions of the client and the different daemons in a
YARN environment:
In a Hadoop cluster, the ResourceManager and NodeManager daemons can be monitored
via specific URLs using a browser. The specific URLs are of the ht-
tp://<serveraddress>:port_number type.
Search WWH ::




Custom Search