Database Reference
In-Depth Information
val
firstRow
=
rows
.
first
()
println
(
firstRow
.
getString
(
0
))
// Field 0 is the name
Example 5-32. Creating a HiveContext and selecting data in Java
import
org.apache.spark.sql.hive.HiveContext
;
import
org.apache.spark.sql.Row
;
import
org.apache.spark.sql.SchemaRDD
;
HiveContext
hiveCtx
=
new
HiveContext
(
sc
);
SchemaRDD
rows
=
hiveCtx
.
sql
(
"SELECT name, age FROM users"
);
Row
firstRow
=
rows
.
first
();
System
.
out
.
println
(
firstRow
.
getString
(
0
));
// Field 0 is the name
We cover loading data from Hive in more detail in
“Apache Hive” on page 170
.
JSON
If you have JSON data with a consistent schema across records, Spark SQL can infer
their schema and load this data as rows as well, making it very simple to pull out the
fields you need. To load JSON data, first create a
HiveContext
as when using Hive.
(No installation of Hive is needed in this case, though—that is, you don't need a
hive-
site.xml
file.) Then use the
HiveContext.jsonFile
method to get an RDD of
Row
objects for the whole file. Apart from using the whole
Row
object, you can also register
this RDD as a table and select specific fields from it. For example, suppose that we
had a JSON file containing tweets in the format shown in
Example 5-33
, one per line.
Example 5-33. Sample tweets in JSON
{
"user"
:
{
"name"
:
"Holden"
,
"location"
:
"San Francisco"
},
"text"
:
"Nice day out today"
}
{
"user"
:
{
"name"
:
"Matei"
,
"location"
:
"Berkeley"
},
"text"
:
"Even nicer here :)"
}
We could load this data and select just the username and text fields as shown in
Examples
5-34
through
5-36
.
Example 5-34. JSON loading with Spark SQL in Python
tweets
=
hiveCtx
.
jsonFile
(
"tweets.json"
)
tweets
.
registerTempTable
(
"tweets"
)
results
=
hiveCtx
.
sql
(
"SELECT user.name, text FROM tweets"
)
Example 5-35. JSON loading with Spark SQL in Scala
val
tweets
=
hiveCtx
.
jsonFile
(
"tweets.json"
)
tweets
.
registerTempTable
(
"tweets"
)
val
results
=
hiveCtx
.
sql
(
"SELECT user.name, text FROM tweets"
)
