Database Reference
In-Depth Information
Driver and Executor Logs
In certain cases users can learn more information from Spark by inspecting the logs
produced directly by the driver program and executors. Logs contain more detailed
traces of anomalous events such as internal warnings or detailed exceptions from
user code. This data can help when you're troubleshooting errors or unexpected
behavior.
The exact location of Spark's logfiles depends on the deployment mode:
• In Spark's Standalone mode, application logs are directly displayed in the stand‐
alone master's web UI. They are stored by default in the
work/
directory of the
Spark distribution on each worker.
• In Mesos, logs are stored in the
work/
directory of a Mesos slave, and accessible
from the Mesos master UI.
• In YARN mode, the easiest way to collect logs is to use YARN's log collection
tool (running
yarn logs -applicationId
<app ID>
) to produce a report con‐
taining logs from your application. This will work only after an application has
fully finished, since YARN must first aggregate these logs together. For viewing
logs of a running application in YARN, you can click through the ResourceMan‐
ager UI to the Nodes page, then browse to a particular node, and from there, a
particular container. YARN will give you the logs associated with output pro‐
duced by Spark in that container. This process is likely to become less round‐
about in a future version of Spark with direct links to the relevant logs.
By default Spark outputs a healthy amount of logging information. It is also possible
to customize the logging behavior to change the logging level or log output to non-
standard locations. Spark's logging subsystem is based on log4j, a widely used Java
logging library, and uses log4j's configuration format. An example log4j configura‐
tion file is bundled with Spark at
conf/log4j.properties.template
. To customize Spark's
logging, first copy the example to a file called
log4j.properties
. You can then modify
behavior such as the root logging level (the threshold level for logging output). By
default, it is
INFO
. For less log output, it can be set to
WARN
or
ERROR
. Once you've
tweaked the logging to match your desired level or format, you can add the
log4j.properties
file using the
--files
flag of
spark-submit
. If you have trouble set‐
ting the log level in this way, make sure that you are not including any JARs that
themselves contain
log4j.properties
files with your application. Log4j works by scan‐
ning the classpath for the first properties file it finds, and will ignore your customiza‐
tion if it finds properties somewhere else first.
