Database Reference
In-Depth Information
Creating a basic streaming application
Next, we will create our first streaming program. We will simply connect to the producer
and print out the contents of each batch. Our streaming code looks like this:
/**
* A simple Spark Streaming app in Scala
*/
object SimpleStreamingApp {
def main(args: Array[String]) {
val ssc = new StreamingContext("local[2]", "First
Streaming App", Seconds(10))
val stream = ssc.socketTextStream("localhost", 9999)
// here we simply print out the first few elements of
each
// batch
stream.print()
ssc.start()
ssc.awaitTermination()
}
}
It looks fairly simple, and it is mostly due to the fact that Spark Streaming takes care of all
the complexity for us. First, we initialized a
StreamingContext
(which is the stream-
ing equivalent of the
SparkContext
we have used so far), specifying similar configura-
tion options that are used to create a
SparkContext
. Notice, however, that here we are
required to provide the batch interval, which we set to 10 seconds.
We then created our data stream using a predefined streaming source,
socketTex-
tStream
, which reads text from a socket host and port and creates a
DStream[String]
. We then called the
print
function on the DStream; this function
prints out the first few elements of each batch.
