Database Reference
In-Depth Information
Resource Manager Failure
Failure of the resource manager is serious, because without it, neither jobs nor task con-
tainers can be launched. In the default configuration, the resource manager is a single
point of failure, since in the (unlikely) event of machine failure, all running jobs fail —
and can't be recovered.
To achieve high availability (HA), it is necessary to run a pair of resource managers in an
active-standby configuration. If the active resource manager fails, then the standby can
take over without a significant interruption to the client.
Information about all the running applications is stored in a highly available state store
(backed by ZooKeeper or HDFS), so that the standby can recover the core state of the
failed active resource manager. Node manager information is not stored in the state store
since it can be reconstructed relatively quickly by the new resource manager as the node
managers send their first heartbeats. (Note also that tasks are not part of the resource man-
ager's state, since they are managed by the application master. Thus, the amount of state
to be stored is therefore much more manageable than that of the jobtracker in MapReduce
1.)
When the new resource manager starts, it reads the application information from the state
store, then restarts the application masters for all the applications running on the cluster.
This does not count as a failed application attempt (so it does not count against
yarn.resourcemanager.am.max-attempts
), since the application did not fail
due to an error in the application code, but was forcibly killed by the system. In practice,
the application master restart is not an issue for MapReduce applications since they recov-
er the work done by completed tasks (as we saw in
Application Master Failure
)
.
The transition of a resource manager from standby to active is handled by a failover con-
troller. The default failover controller is an automatic one, which uses ZooKeeper leader
election to ensure that there is only a single active resource manager at one time. Unlike in
HDFS HA (see
HDFS High Availability
), the failover controller does not have to be a
standalone process, and is embedded in the resource manager by default for ease of con-
figuration. It is also possible to configure manual failover, but this is not recommended.
Clients and node managers must be configured to handle resource manager failover, since
there are now two possible resource managers to communicate with. They try connecting
to each resource manager in a round-robin fashion until they find the active one. If the act-
ive fails, then they will retry until the standby becomes active.
