Database Reference
In-Depth Information
Figure 14-2. Flume agent with a spooling directory source and fanning out to an HDFS sink and a log-
ger sink
Delivery Guarantees
Flume uses a separate transaction to deliver each batch of events from the spooling direct-
ory source to each channel. In this example, one transaction will be used to deliver to the
channel feeding the HDFS sink, and then another transaction will be used to deliver the
same batch of events to the channel for the logger sink. If either of these transactions fails
(if a channel is full, for example), then the events will not be removed from the source,
and will be retried later.
In this case, since we don't mind if some events are not delivered to the logger sink, we
can designate its channel as an
optional
channel, so that if the transaction associated with
it fails, this will not cause events to be left in the source and tried again later. (Note that if
the agent fails before
both
channel transactions have committed, then the affected events
will be redelivered after the agent restarts — this is true even if the uncommitted channels
are marked as optional.) To do this, we set the
selector.optional
property on the
source, passing it a space-separated list of channels:
agent1.sources.source1.selector.optional
=
channel1b
