Database Reference
In-Depth Information
eirmod tempor
invidunt ut labore et dolore magna aliquyam]
Connecting Storm to Kafka
With the introduction of Kafka 0.8, the integration between Kafka and
Storm in 0.9 is in a state of flux. The
storm-kafka
spout implementation
available in the
storm-contrib
library is currently targeted to version
0.7.2 of Kafka. A version of the Kafka spout designed for use with Kafka
0.8 is available through another project called
storm-kafka-0.8-plus
(
https://github.com/wurstmeister/storm-kafka-0.8-plus
)
. The project is
also available via
clojars.org
, the same as Storm itself for inclusion in a
Maven
pom.xml
file. It is included in a project using the following artifact:
<dependency>
<groupId>net.wurstmeister.storm</groupId>
<artifactId>storm-kafka-0.8-plus</artifactId>
<version>0.2.0</version>
</dependency>
To use the
Spout
, a
KafkaConfig
object is first configured to point to the
appropriate set of brokers and the desired topic. For example, the following
code obtains the list of brokers from ZooKeeper. It then attaches itself to the
wikipedia-raw
topic using the
storm
Consumer Group:
BrokerHosts hosts =
new
ZkHosts("localhost");
SpoutConfig config =
new
SpoutConfig(hosts,
"wikipedia-raw", "", "storm");
config.scheme =
new
SchemeAsMultiScheme(
new
StringScheme());
The
config.scheme
entry tells the spout to interpret incoming messages
as strings rather than some more exotic entry. Other encodings would
require a different scheme implementation specific to that format.
