Database Reference
In-Depth Information
spooldir
source and an Avro sink connected by a file channel. The
agent2
agent runs
in the second tier, and has an Avro source that listens on the port that
agent1
's Avro sink
sends events to. The sink for
agent2
uses the same HDFS sink configuration from
Notice that since there are two file channels running on the same machine, they are con-
figured to point to different data and checkpoint directories (they are in the user's home
directory by default). This way, they don't try to write their files on top of one another.
Example 14-4. A two-tier Flume configuration using a spooling directory source and an
HDFS sink
# First-tier agent
agent1.sources
=
source1
agent1.sinks
=
sink1
agent1.channels
=
channel1
agent1.sources.source1.channels
=
channel1
agent1.sinks.sink1.channel
=
channel1
agent1.sources.source1.type
=
spooldir
agent1.sources.source1.spoolDir
=
/tmp/spooldir
agent1.sinks.sink1.type
=
avro
agent1.sinks.sink1.hostname
=
localhost
agent1.sinks.sink1.port
=
10000
agent1.channels.channel1.type
=
file
agent1.channels.channel1.checkpointDir
=
/tmp/agent1/file-channel/
checkpoint
agent1.channels.channel1.dataDirs
=
/tmp/agent1/file-channel/data
# Second-tier agent
agent2.sources
=
source2
agent2.sinks
=
sink2
agent2.channels
=
channel2
agent2.sources.source2.channels
=
channel2
agent2.sinks.sink2.channel
=
channel2
agent2.sources.source2.type
=
avro
agent2.sources.source2.bind
=
localhost
agent2.sources.source2.port
=
10000
