Database Reference
In-Depth Information
Table 14-1. Flume components
Category Component Description
Source
Avro
Listens on a port for events sent over Avro RPC by an Avro sink or the Flume
SDK.
Exec
Runs a Unix command (e.g.,
tail -F
/path/to/file
) and converts lines
read from standard output into events. Note that this source cannot guarantee
delivery of events to the channel; see the spooling directory source or the
Flume SDK for better alternatives.
HTTP
Listens on a port and converts HTTP requests into events using a pluggable
handler (e.g., a JSON handler or binary blob handler).
JMS
Reads messages from a JMS queue or topic and converts them into events.
Netcat
Listens on a port and converts each line of text into an event.
Sequence
generator
Generates events from an incrementing counter. Useful for testing.
Spooling
directory
Reads lines from files placed in a spooling directory and converts them into
events.
Syslog
Reads lines from syslog and converts them into events.
Thrift
Listens on a port for events sent over Thrift RPC by a Thrift sink or the Flume
SDK.
Twitter
Connects to Twitter's streaming API (1% of the firehose) and converts tweets
into events.
Sink
Avro
Sends events over Avro RPC to an Avro source.
Elasticsearch Writes events to an Elasticsearch cluster using the Logstash format.
File roll
Writes events to the local filesystem.
HBase
Writes events to HBase using a choice of serializer.
HDFS
Writes events to HDFS in text, sequence file, Avro, or a custom format.
IRC
Sends events to an IRC channel.
Logger
Logs events at
INFO
level using SLF4J. Useful for testing.
Morphline
(Solr)
Runs events through an in-process chain of Morphline commands. Typically
used to load data into Solr.
Null
Discards all events.
Thrift
Sends events over Thrift RPC to a Thrift source.
