Database Reference
In-Depth Information
Classpath Integration
If plug-in integration is not possible for some reason, Flume can also use
a more standard classpath-based approach. To use this approach, use the
FLUME_CLASSPATH
environment variable when starting the Flume Agent
to add libraries to the Flume agent for later use.
Running Flume Agents
Starting the Flume agent is done using the
flume-ng
script found in the
Flume binary directory. The startup script requires a name, a configuration
directory and a configuration properties file (some configuration options
may reference other files in the configuration directory):
$ ./bin/flume-ng agent --conf conf \
> -f conf/agent.properties --name agent-1
The
agent
command tells Flume that it is starting an agent. The
--conf
specifies a configuration directory, and the
-f
switch specifies the
properties file containing the agent definitions. The
-name
switch tells
Flume to start the agent named
agent-1
. After the agent is started, it
begins processing events according to its configuration.
Conclusion
This chapter has discussed two systems for handling data flow in a
distributed environment. Both Kafka and Flume are high-performance
systems with sufficient scalability to support real-time streaming. Kafka is
directed toward users who are building applications from scratch, giving
them the freedom to directly integrate a robust data motion system. Flume
can also be used by new applications, but its design makes it well suited to
environments that have existing applications that need to be federated into
a single processing environment.
The temptation at this point is to consider these two systems to be somehow
mutually exclusive: Kafka for new things and Flume for “legacy” things.
This is simply not the case. The two systems are complementary to each
other,solvingdifferentusagepatterns.Inacomplicatedenvironment,Kafka
might be used for the bulk data motion within the system, handling the
bulk logging and collecting data from outlying data sources. It might also be
