Testing and Deployment: Quality Is King - Deploying and Managing a Cloud Infrastructure - page 316

Listing 10.4: Collector flume-conf.properties for 10.0.0.3
Collector configuration (flume-conf.properties on 10.0.0.3):
# The configuration file needs to define the sources,
# the channels and the sinks.
# Sources, channels and sinks are defined per agent,
# in this case called 'agent'
agent.sources = avro-collection-source
agent.channels = memoryChannel
agent.sinks = hdfs-sink
# For each one of the sources, the type is defined
agent.sources.avro-collection-source.type = avro
agent.sources.avro-collection-source.bind = 10.0.0.3
agent.sources.avro-collection-source.port = 60000
# The channel can be defined as follows.
agent.sources.avro-collection-source.channels = memoryChannel
# Each sink's type must be defined
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://10.0.10.1:8020/flume
#Specify the channel the sink should use
agent.sinks.hdfs-sink.channel = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.memoryChannel.capacity = 10000
Execution
Setting up the whole system requires three steps. Note that the configuration files are
responsible for how a node behaves (i.e., as an agent or collector).
1. Compiling the Java code and executing two instances, one each on each agent node.
$ javac RSSReader.java
$ java -cp /root/RSSReader RSSReader > /var/log/flume-ng/source.txt &