Database Reference
In-Depth Information
NOTE
There is another type of processor,
failover
, that instead of load balancing events across sinks uses a
preferred sink if it is available, and fails over to another sink in the case that the preferred sink is down.
The failover sink processor maintains a priority order for sinks in the group, and attempts delivery in or-
der of priority. If the sink with the highest priority is unavailable the one with the next highest priority is
tried, and so on. Failed sinks are blacklisted for an increasing timeout period (up to a maximum period of
30 seconds, controlled by
processor.maxpenalty
).
Example 14-6. Flume configuration for second-tier agent in a load balancing scenario
agent2a.sources
=
source2a
agent2a.sinks
=
sink2a
agent2a.channels
=
channel2a
agent2a.sources.source2a.channels
=
channel2a
agent2a.sinks.sink2a.channel
=
channel2a
agent2a.sources.source2a.type
=
avro
agent2a.sources.source2a.bind
=
localhost
agent2a.sources.source2a.port
=
10000
agent2a.sinks.sink2a.type
=
hdfs
agent2a.sinks.sink2a.hdfs.path
=
/tmp/flume
agent2a.sinks.sink2a.hdfs.filePrefix
=
events-a
agent2a.sinks.sink2a.hdfs.fileSuffix
=
.log
agent2a.sinks.sink2a.hdfs.fileType
=
DataStream
agent2a.channels.channel2a.type
=
file
The configuration for
agent2b
is the same, except for the Avro source port (since we are
running the examples on localhost) and the file prefix for the files created by the HDFS
sink. The file prefix is used to ensure that HDFS files created by second-tier agents at the
same time don't collide.
In the more usual case of agents running on different machines, the hostname can be used
to make the filename unique by configuring a host interceptor (see
Table 14-1
) and includ-
ing the
%{host}
escape sequence in the file path, or prefix:
agent2.sinks.sink2.hdfs.filePrefix
=
events-%{host}
A diagram of the whole system is shown in
Figure 14-6
.
