Database Reference
In-Depth Information
Figure 10-2. DStream as a continuous series of RDDs
You can create DStreams either from external input sources, or by applying
transfor‐
mations
to other DStreams. DStreams support many of the transformations that you
saw on RDDs in
Chapter 3
. Additionally, DStreams also have new “stateful” transfor‐
mations that can aggregate data across time. We will discuss these in the next section.
In our simple example, we created a DStream from data received through a socket,
and then applied a
filter()
transformation to it. This internally creates RDDs as
shown in
Figure 10-3
.
Figure 10-3. DStreams and transformation of Examples
10-4
through
10-8
If you run
Example 10-8
, you should see something similar to the output in
Example 10-9
.
Example 10-9. Log output from running
Example 10-8
-------------------------------------------
Time: 1413833674000 ms
-------------------------------------------
71.19.157.174 - - [24/Sep/2014:22:26:12 +0000] "GET /error78978 HTTP/1.1" 404 505
...
-------------------------------------------
Time: 1413833675000 ms
-------------------------------------------
71.19.164.174 - - [24/Sep/2014:22:27:10 +0000] "GET /error78978 HTTP/1.1" 404 505
...
