Spark SQL - Learning Spark - page 164

Note that if you don't have an existing Hive installation, Spark SQL will create its
own Hive metastore (metadata DB) in your program's work directory, called meta
store_db . In addition, if you attempt to create tables using HiveQL's CREATE TABLE
statement (not CREATE EXTERNAL TABLE ), they will be placed in the /user/hive/ware‐
house directory on your default filesystem (either your local filesystem, or HDFS if
you have a hdfs-site.xml on your classpath).
Using Spark SQL in Applications
The most powerful way to use Spark SQL is inside a Spark application. This gives us
the power to easily load data and query it with SQL while simultaneously combining
it with “regular” program code in Python, Java, or Scala.
To use Spark SQL this way, we construct a HiveContext (or SQLContext for those
wanting a stripped-down version) based on our SparkContext. This context provides
additional functions for querying and interacting with Spark SQL data. Using the
HiveContext, we can build SchemaRDDs, which represent our structure data, and
operate on them with SQL or with normal RDD operations like map() .
Initializing Spark SQL
To get started with Spark SQL we need to add a few imports to our programs, as
shown in Example 9-2 .
Example 9-2. Scala SQL imports
// Import Spark SQL
import org.apache.spark.sql.hive.HiveContext
// Or if you can't have the hive dependencies
import org.apache.spark.sql.SQLContext
Scala users should note that we don't import HiveContext._ , like we do with the
SparkContext, to get access to implicits. These implicits are used to convert RDDs
with the required type information into Spark SQL's specialized RDDs for querying.
Instead, once we have constructed an instance of the HiveContext we can then
import the implicits by adding the code shown in Example 9-3 . The imports for Java
and Python are shown in Examples 9-4 and 9-5 , respectively.
Example 9-3. Scala SQL implicits
// Create a Spark SQL HiveContext
val hiveCtx = ...
// Import the implicit conversions
import hiveCtx._