Databases Reference
In-Depth Information
1.3. Installing Specialized Connectors
Problem
Some database systems provide special connectors, which are not part of the Sqoop
distribution, and these take advantage of advanced database features. If you want to take
advantage of these optimizations, you will need to individually download and install
those specialized connectors.
Solution
On the node running Sqoop, you can install the specialized connectors anywhere on
the local filesystem. If you plan to run Sqoop from multiple nodes, you have to install
the connector on all of those nodes. To be clear, you do not have to install the connector
on all nodes in your cluster, as Sqoop will automatically propagate the appropriate JARs
as needed throughout your cluster.
In addition to installing the connector JARs on the local filesystem, you also need to
register them with Sqoop. First, create a directory
manager.d
in the Sqoop configuration
directory (if it does not exist already). The configuration directory might be in a different
location, based on how you've installed Sqoop. With packages, it's usually in the
/etc/
sqoop
directory, and with tarballs, it's usually in the
conf/
directory. Then, inside this
directory, you need to create a file (naming it after the connector is a recommended
best practice) that contains the following line:
connector.fully.qualified.class.name=/full/path/to/the/jar
You can find the name of the fully qualified class in each connector's documentation.
Discussion
A significant strength of Sqoop is its ability to work with all major and minor database
systems and enterprise data warehouses. To abstract the different behavior of each sys‐
tem, Sqoop introduced the concept of connectors: all database-specific operations are
delegated from core Sqoop to the specialized connectors. Sqoop itself bundles many
such connectors; you do not need to download anything extra in order to run Sqoop.
The most general connector bundled with Sqoop is the Generic JDBC Connector that
utilizes only the JDBC interface. This will work with every JDBC-compliant database
system. In addition to this generic connector, Sqoop also ships with specialized con‐
nectors for MySQL, Oracle, PostgreSQL, Microsoft SQL Server, and DB2, which utilize
special properties of each particular database system. You do not need to explicitly select
the desired connector, as Sqoop will automatically do so based on your JDBC URL.
