Database Reference
In-Depth Information
Table 8-1. Common Spark configuration values
Option(s)
Default
Explanation
512m
Amount of memory to use per executor process, in
the same format as JVM memory strings (e.g.,
512m, 2g). See
“Hardware Provisioning” on page 158
for more detail on this option.
spark.executor.memory
(--executor-memory)
Configurations for bounding the number of cores
used by the application. In YARN mode
spark.executor.cores
will assign a specific
number of cores to each executor. In standalone and
Mesos modes, you can upper-bound the total
number of cores across all executors using
spark.cores.max
. Refer to
“Hardware
Provisioning” on page 158
for more detail.
spark.executor.cores(--
executor-cores)
spark.cores.max(--total-
executor-cores)
1
(none)
Setting to
true
will enable speculative execution
of tasks. This means tasks that are running slowly
will have a second copy launched on another node.
Enabling this can help cut down on straggler tasks
in large clusters.
spark.speculation
false
An internal timeout used for tracking the liveness of
executors. For jobs that have long garbage
collection pauses, tuning this to be 100 seconds (a
value of
100000
) or higher can prevent thrashing.
In future versions of Spark this may be replaced
with a general timeout setting, so check current
documentation.
spark.storage.blockMana
gerTimeoutIntervalMs
45000
(empty)
These three options allow you to customize the
launch behavior of executor JVMs. The three flags
add extra Java options, classpath entries, or path
entries for the JVM library path. These parameters
should be specified as strings (e.g.,
spark.executor.extraJavaOptions="-
XX:+PrintGCDetails-XX:+PrintGCTi
meStamps"
). Note that while this allows you to
manually augment the executor classpath, the
recommended way to add dependencies is through
the
--jars
flag to
spark-submit
(not using
this option).
spark.executor.extraJa
vaOptions
spark.executor.extraC
lassPath
spark.executor.extraLi
braryPath
