Testing and Deployment: Quality Is King - Deploying and Managing a Cloud Infrastructure - page 314

Listing 10.2 and Listing 10.3 show sample configurations for agents (10.0.0.1 and
10.0.0.2). Listing 10.4 shows the configuration for the collector (10.0.0.3). The purpose of
the configuration files is to define semantics for source, channel, and sink. For each source
type, we also need to define type, command, standard error behavior, and failure options.
For each channel, we need to define the channel type. The channel type, capacity (maxi-
mum number of events stored in the channel), and transaction capacity (maximum num-
ber of events the channel will take from a source or give to a sink per transaction) have to
be defined as well. Similarly, for each sink type, we need to define type, hostname (IP of
the recipient of the event), and port. In case of an HDFS sink, the path to the HDFS head
name node has to be provided.
Listing 10.2: Agent 1 flume-conf.properties for 10.0.0.1
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent.sources = reader
agent.channels = memoryChannel
agent.sinks = avro-forward-sink
# For each one of the sources, the type is defined
agent.sources.reader.type = exec
agent.sources.reader.command = tail -f /var/log/flume-ng/source.txt
# stderr is simply discarded, unless logStdErr=true
# If the process exits for any reason, the source also exits and will produce no
# further data.
agent.sources.reader.logStdErr = true
agent.sources.reader.restart = true
# The channel can be defined as follows.
agent.sources.reader.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = 10.0.0.3
agent.sinks.avro-forward-sink.port = 60000
#Specify the channel the sink should use
agent.sinks.avro-forward-sink.channel = memoryChannel
# Each channel's type is defined.