Database Reference
In-Depth Information
NEAR-REAL-TIME INDEXING
Indexing events for search is a good example of where fan out is used in practice. A single source of
events is sent to both an HDFS sink (this is the main repository of events, so a required channel is used)
and a Solr (or Elasticsearch) sink, to build a search index (using an optional channel).
The
MorphlineSolrSink
extracts fields from Flume events and transforms them into a Solr docu-
ment (using a Morphline configuration file), which is then loaded into a live Solr search server. The pro-
cess is called
near real time
since ingested data appears in search results in a matter of seconds.
Replicating and Multiplexing Selectors
In normal fan-out flow, events are replicated to all channels — but sometimes more se-
lective behavior might be desirable, so that some events are sent to one channel and others
to another. This can be achieved by setting a
multiplexing
selector on the source, and de-
fining routing rules that map particular event header values to channels. See the
Flume
User Guide
for configuration details.
