Database Reference
In-Depth Information
In addition to remote datacenter mirroring, the mirror facilities are also
useful for development. A remote cluster can simply be mirrored into the
development cluster to allow development using production data without
risking production environments.
Configuring a Kafka Environment
This section describes how to start a Kafka environment. On development
systems, a single broker is usually sufficient for testing whereas a larger
system should probably have a minimum of three to five brokers depending
on replication requirements. On the top end, quite large installations are
possible. Chapter 5, “Processing Streaming Data,” covers a tool called
Samza, which uses Kafka for data processing in the same way that Hadoop
Map-Reduce uses its distributed filesystem. In this instance, there could be
dozens of brokers in the cluster that are all running Kafka.
This section covers everything needed to get Kafka up and running. There
is also a small section on developing multi-broker applications on a single
machine by using separate configurations.
Installing Kafka
Installing Kafka is simply a matter of unpacking the current version of
Kafka, 0.8.0 at the time of writing, and configuring it. Kafka itself is largely
written in a language called Scala, which has several versions. To avoid
confusion between versions, the Kafka developers choose to include the
build version of Scala in the archive name. This is mostly important for
developers who are trying to use the libraries included in the archive for
development. Currently, Kafka's binary distribution is built using Scala
2.8.0, so the archive is named
kafka_2.8.0-0.0.tar.gz
. You can
download it from
http://kafka.apache .org
and then unpack it:
$ tar xvfz ~/Downloads/kafka_2.8.0-0.8.0.tar.gz
x kafka_2.8.0-0.8.0/
x kafka_2.8.0-0.8.0/libs/
x kafka_2.8.0-0.8.0/libs/slf4j-api-1.7.2.jar
x kafka_2.8.0-0.8.0/libs/zkclient-0.3.jar
x kafka_2.8.0-0.8.0/libs/scala-compiler.jar
x kafka_2.8.0-0.8.0/libs/snappy-java-1.0.4.1.jar
x kafka_2.8.0-0.8.0/libs/metrics-core-2.2.0.jar
x kafka_2.8.0-0.8.0/libs/metrics-annotation-2.2.0.jar
