Database Reference
In-Depth Information
--executor-cores 1 \
--executor-memory 2g
The YARN resource manager address is not specified in the master URL (unlike when us-
ing the standalone or Mesos cluster managers), but is picked up from Hadoop configura-
tion in the directory specified by the
HADOOP_CONF_DIR
environment variable.
YARN cluster mode
In YARN cluster mode, the user's driver program runs in a YARN application master pro-
cess. The
spark-submit
command is used with a master URL of
yarn-cluster
:
%
spark-submit --master yarn-cluster ...
All other parameters, like
--num-executors
and the application JAR (or Python file),
are the same as for YARN client mode (use
spark-submit --help
for usage).
The
spark-submit
client will launch the YARN application (step 1 in
Figure 19-4
)
, but it
doesn't run any user code. The rest of the process is the same as client mode, except the
application master starts the driver program (step 3b) before allocating resources for ex-
ecutors (step 4).
