Tuning and Debugging Spark - Learning Spark - page 143

Example 8-4. Setting configuration values at runtime using flags
$ bin/spark-submit \
--class com.example.MyApp \
--master local [ 4 ] \
--name "My Spark App" \
--conf spark.ui.port = 36000 \
myApp.jar
spark-submit also supports loading configuration values from a file. This can be use‐
ful to set environmental configuration, which may be shared across multiple users,
such as a default master. By default, spark-submit will look for a file called conf/
spark-defaults.conf in the Spark directory and attempt to read whitespace-delimited
key/value pairs from this file. You can also customize the exact location of the file
using the --properties-file flag to spark-submit , as you can see in Example 8-5 .
Example 8-5. Setting configuration values at runtime using a defaults file
$ bin/spark-submit \
--class com.example.MyApp \
--properties-file my-config.conf \
myApp.jar
## Contents of my-config.conf ##
spark.master local [ 4 ]
spark.app.name "My Spark App"
spark.ui.port 36000
The SparkConf associated with a given application is immutable
once it is passed to the SparkContext constructor. That means that
all configuration decisions must be made before a SparkContext is
instantiated.
In some cases, the same configuration property might be set in multiple places. For
instance, a user might call setAppName() directly on a SparkConf object and also pass
the --name flag to spark-submit . In these cases Spark has a specific precedence order.
The highest priority is given to configurations declared explicitly in the user's code
using the set() function on a SparkConf object. Next are flags passed to spark-
submit , then values in the properties file, and finally default values. If you want to
know which configurations are in place for a given application, you can examine a list
of active configurations displayed through the application web UI discussed later in
this chapter.
Several common configurations were listed in Table 7-2 . Table 8-1 outlines a few
additional configuration options that might be of interest. For the full list of configu‐
ration options, see the Spark documentation .