Database Reference
In-Depth Information
The shell prompt should appear within a few seconds. When the shell starts, you will
notice a lot of log messages. You may need to press Enter once to clear the log output
and get to a shell prompt.
Figure 2-1
shows what the PySpark shell looks like when
you open it.
Figure 2-1. The PySpark shell with default logging output
You may find the logging statements that get printed in the shell distracting. You can
control the verbosity of the logging. To do this, you can create a file in the
conf
direc‐
tory called
log4j.properties
. The Spark developers already include a template for this
file called
log4j.properties.template
. To make the logging less verbose, make a copy of
conf/log4j.properties.template
called
conf/log4j.properties
and find the following line:
log4j.rootCategory
=
INFO, console
Then lower the log level so that we show only the WARN messages, and above by
changing it to the following:
log4j.rootCategory
=
WARN, console
When you reopen the shell, you should see less output (
Figure 2-2
).
