Database Reference
In-Depth Information
.
map
(
rec
=>
(
rec
(
0
).
toInt
,
rec
(
1
).
toInt
))
.
reduceByKey
((
a
,
b
)
=>
Math
.
max
(
a
,
b
))
.
saveAsTextFile
(
args
(
1
))
}
}
When running a standalone program, we need to create the
SparkContext
since there
is no shell to provide it. We create a new instance with a
SparkConf
, which allows us to
pass various Spark properties to the application; here we just set the application name.
There are a couple of other minor changes. The first is that we've used the command-line
arguments to specify the input and output paths. We've also used method chaining to
avoid having to create intermediate variables for each RDD. This makes the program
more compact, and we can still view the type information for each transformation in the
Scala IDE if needed.
NOTE
Not all the transformations that Spark defines are available on the
RDD
class itself. In this case,
re-
ducebyKey()
(which acts only on RDDs of key-value pairs) is actually defined in the
PairRDDFunctions
class, but we can get Scala to implicitly convert
RDD[(Int, Int)]
to
PairRDDFunctions
with the following import:
import
org.apache.spark.SparkContext._
This imports various implicit conversion functions used in Spark, so it is worth including in programs as
a matter of course.
This time we use
spark-submit
to run the program, passing as arguments the application
JAR containing the compiled Scala program, followed by our program's command-line
arguments (the input and output paths):
%
spark-submit --class MaxTemperature --master local \
spark-examples.jar input/ncdc/micro-tab/sample.txt output
%
cat output/part-*
(1950,22)
(1949,111)
We also specified two options:
--class
to tell Spark the name of the application class,
and
--master
to specify where the job should run. The value
local
tells Spark to run
everything in a single JVM on the local machine. We'll learn about the options for run-
ning on a cluster in
Executors and Cluster Managers
.
Next, let's see how to use other lan-
guages with Spark, starting with Java.
