Database Reference
In-Depth Information
Fan Out
Fan out is the term for delivering events from one source to multiple channels, so they
reach multiple sinks. For example, the configuration in Example 14-3 delivers events to
both an HDFS sink ( sink1a via channel1a ) and a logger sink ( sink1b via chan-
nel1b ).
Example 14-3. Flume configuration using a spooling directory source, fanning out to an
HDFS sink and a logger sink
agent1.sources = source1
agent1.sinks = sink1a sink1b
agent1.channels = channel1a channel1b
agent1.sources.source1.channels = channel1a channel1b
agent1.sinks.sink1a.channel = channel1a
agent1.sinks.sink1b.channel = channel1b
agent1.sources.source1.type = spooldir
agent1.sources.source1.spoolDir = /tmp/spooldir
agent1.sinks.sink1a.type = hdfs
agent1.sinks.sink1a.hdfs.path = /tmp/flume
agent1.sinks.sink1a.hdfs.filePrefix = events
agent1.sinks.sink1a.hdfs.fileSuffix = .log
agent1.sinks.sink1a.hdfs.fileType = DataStream
agent1.sinks.sink1b.type = logger
agent1.channels.channel1a.type = file
agent1.channels.channel1b.type = memory
The key change here is that the source is configured to deliver to multiple channels by set-
ting agent1.sources.source1.channels to a space-separated list of channel
names, channel1a and channel1b . This time, the channel feeding the logger sink
( channel1b ) is a memory channel, since we are logging events for debugging purposes
and don't mind losing events on agent restart. Also, each channel is configured to feed one
sink, just like in the previous examples. The flow is illustrated in Figure 14-2 .
Search WWH ::




Custom Search