Databases Reference
In-Depth Information
Applicaton submits requests to YARN
YARN ASM negotiates for resources and launches MR Application Manager
MR AM handshakes with ASM
MR AM completes splits and sets up API and maps, sets YA RN scheduler
MR AM runs the job and monitors resources, negotiates for resources as needed
MR AM monitors job to completion and executes cleanup and Output Committer
FIGURE 4.12
YARN execution flow.
What this means to any application is the memory slots required to run a job can be accessed
from any node, depending on the availability of memory. This provides simple chunkable scalability
especially in a cluster configuration. In classic Hadoop MapReduce the cluster is not artificially seg-
regated into Map and Reduce slots, and the application jobs are bottlenecked on Reduce slots limiting
scalability in job execution in the dataflow (
Figure 4.12
).
Comparison between MapReduce v1 and v2
Presented here is a simple comparison between the two releases of MapReduce:
●
Classic MapReduce:
●
Job request submitted to JobTracker.
●
Jobtracker manages the execution with tasks.
●
Resources are allocated on availability basis, some jobs get more and others less.
●
No scalable resource allocation across a cluster.
●
Multiple single points of failure.
●
YARN:
●
Application executed by YARN.
●
Resources negotiated and allocated prior to job execution.
●
Map-based resource request setup for the entire job.
●
Resource monitor tracks usage and requests additional resources as needed from across a
cluster in a clustered setup.
●
Job completion and cleanup tasks are executed.
