Moving Data - Big Data Made Easy: A Working Guide to the Complete Hadoop Toolset - page 172

18 # ----------------------------------------------------------------------
19 # define agent source
20 # ----------------------------------------------------------------------
21
22 agent1.sources.source1.type = exec
23 agent1.sources.source1.command = tail -F /var/log/messages
24 agent1.sources.source1.channels = channel1
25
26 # ----------------------------------------------------------------------
27 # define agent sink
28 # ----------------------------------------------------------------------
29
30 agent1.sinks.sink1.type = hdfs
31 agent1.sinks.sink1.hdfs.path = hdfs://hc1nn/flume/messages
32 agent1.sinks.sink1.hdfs.rollInterval = 0
33 agent1.sinks.sink1.hdfs.rollSize = 1000000
34 agent1.sinks.sink1.hdfs.batchSize = 100
35 agent1.sinks.sink1.channel = channel1
As already defined in the agent execution script, the Flume agent name in this example is agent1. Lines 5 to 7
define the names of the source, channel, and sink.
5 agent1.sources = source1
6 agent1.channels = channel1
7 agent1.sinks = sink1
The channel (channel1) is described between lines 13 and 16. Line 13 specifies that the channel type will be a
file. Line 14 indicates that the maximum capacity of the channel will be 2 million events. Line 15, in milliseconds,
indicates the time between checkpoints. Line 16 specifies the maximum channel file size in bytes.
13 agent1.channels.channel1.type = FILE
14 agent1.channels.channel1.capacity = 2000000
15 agent1.channels.channel1.checkpointInterval = 60000
16 agent1.channels.channel1.maxFileSize = 10737418240
The configuration file lines (22 to 24 ) show how the Flume data source source1 is defined.
22 agent1.sources.source1.type = exec
23 agent1.sources.source1.command = tail -F /var/log/messages
24 agent1.sources.source1.channels = channel1
In this example, I may need to ensure that the Linux account I am using to run this Flume job has access to read the
log file /var/log/messages. Therefore, I grant access using the root account as follows: I use the Linux su (switch user)
command to change the user ID to root. Then I use the Linux chmod command to grant global read privileges while
maintaining current access. The two Linux ls command listings show that extra access has been granted:
su -
ls -l /var/log/messages
-rw------- 1 root root 410520 Nov 22 09:20 /var/log/messages
chmod 644 /var/log/messages
ls -l /var/log/messages
-rw-r--r-- 1 root root 410520 Nov 22 09:25 /var/log/messages
exit