Database Reference
In-Depth Information
</executions>
</plugin>
</plugins>
</build>
</project>
This project declares two transitive dependencies:
jopt-simple
, a Java library to per‐
form option parsing, and
joda-time
, a library with utilities for time and date conver‐
sion. It also depends on Spark, but Spark is marked as
provided
to ensure that Spark
is never packaged with the application artifacts. The build includes the
maven-shade-
plugin
to create an uber JAR containing all of its dependencies. You enable this by
asking Maven to execute the
shade
goal of the plug-in every time a
package
phase
occurs. With this build configuration, an uber JAR is created automatically when
mvn
package
is run (see
Example 7-6
).
Example 7-6. Packaging a Spark application built with Maven
$
mvn package
# In the target directory, we'll see an uber JAR and the original package JAR
$
ls target/
example-build-1.0.jar
original-example-build-1.0.jar
# Listing the uber JAR will reveal classes from dependency libraries
$
jar tf target/example-build-1.0.jar
...
joptsimple/HelpFormatter.class
...
org/joda/time/tz/UTCProvider.class
...
# An uber JAR can be passed directly to spark-submit
$
/path/to/spark/bin/spark-submit --master
local
... target/example-build-1.0.jar
A Scala Spark Application Built with sbt
sbt is a newer build tool most often used for Scala projects. sbt assumes a similar
project layout to Maven. At the root of your project you create a build file called
build.sbt
and your source code is expected to live in
src/main/scala
. sbt build files are
written in a configuration language where you assign values to specific keys in order
to define the build for your project. For instance, there is a key called
name
, which
contains the project name, and a key called
libraryDependencies
, which contains a
list of dependencies of your project.
Example 7-7
gives a full sbt build file for a simple
application that depends on Spark along with a few other third-party libraries. This
build file works with sbt 0.13. Since sbt evolves quickly, you may want to read its
most recent documentation for any formatting changes in the build file format.
