Database Reference
In-Depth Information
// Convert the MapWritable[Text, Text] to Map[String, String]
val
tweets
=
currentTweets
.
map
{
case
(
key
,
value
)
=>
mapWritableToInput
(
value
)
}
Compared to some of our other connectors, this is a bit convoluted, but it serves as a
useful reference for how to work with these types of connectors.
On the write side, Elasticsearch can do mapping inference, but this
can occasionally infer the types incorrectly, so it can be a good idea
to
explicitly set a mapping
if you are storing data types other than
strings.
Conclusion
With the end of this chapter you should now be able to get your data into Spark to
work with and store the result of your computation in a format that is useful for you.
We have examined a number of different formats we can use for our data, as well as
compression options and their implications on how data can be consumed. Subse‐
quent chapters will examine ways to write more effective and powerful Spark pro‐
grams now that we can load and save large datasets.
