Database Reference
In-Depth Information
agent_1.source.source-1.restartThrottle= 1000
Normally, the Exec source only reads from the standard input. It can also
read events from the standard error stream by setting the
logStdErr
property:
agent_1.source.source-1.logStdErr= true
To improve performance, the Exec source reads a batch of lines before
sending them on to their assigned channel(s). The number of lines, which
defaults to 20, is controlled with the
batchSize
property:
agent_1.source.source-1.batchSize= 20
Unidirectional Sources Do Not Support
“Backpressure”
If at all possible, avoid sources like the Exec source in production
environments. Although sources like the Exec source are very tempting,
they render any reliability guarantees made by the Flume framework
meaningless. The framework has no mechanism for communicating
with the event generator. It cannot report channels that have filled or
otherwise become unavailable. The end result being that the generator
will continue to blindly write to the source, and data is potentially lost.
Fortunately, the most common use case is to emulate the Tail source
(which was removed for a reason), which can usually be better
implemented using the Spool Directory source.
Spool Directory Source
The most common method for data recording for services that do not make
use of a data motion system is logging to the file system. Early versions
of Flume tried to take advantage of this by implementing a Tail source
that allowed Flume to watch the logs as they were being written and move
them into Flume. Unfortunately, there are a number of problems with this
approach if the goal is to provide a reliable data motion pipeline. The Spool
