Database Reference
In-Depth Information
This section introduces the Flume framework. The framework itself ships
with a number of out-of-the-box components that can be used to construct
immediately useful infrastructure. These basic components and their usage
are covered in fair detail. It is also possible to develop custom components,
which is usually mostly done to handle business-specific data formats or
processing needs. Flume, like Kafka, is developed on top of the JVM, so
extending the framework is fairly straightforward.
The Flume Agent
Flume uses a network-flow model as its primary organizing feature, which
is encapsulated into a single machine known as an agent. These Agent
machines are roughly analogous to Kafka's broker and minimally consist of
three components: a source, a channel, and a sink. An agent can have any
number of these basic flows, organized as shown in
Figure 4.2
.
As shown in
Figure 4.2
, the source is the start of any flow on an agent.
Sources are components that either retrieve or receive data from an outside
source. In most cases, this is data from an application. For example, a web
server may produce filesystem logs. To retrieve them, a source that can tail
log files is used as part of an agent installed on the machine running the web
server.
After the source has received data, it transfers this data to one or more
channels. Channels define the persistence and transaction semantics.
Depending on the channel used, the agent can store its data to disk or a
database, or it can keep it in memory. This allows different performance and
reliability characteristics to be selected for each agent.
A source may also choose to write its data to multiple channels, as shown in
Figure 4.3
. This is typically used to multiplex data for better load balancing
and processing parallelism. It can also be used to replicate data or provide
multiple data paths for the entire stream. To define how this happens, the
sourcedefinition usesaselector.Thedefaultselectorisareplication selector
