Database Reference
In-Depth Information
This channel is set by assigning it a type of
memory
. In addition, there are
several optional parameters that can be set. The most useful of these are the
capacity
and
transactionCapacity
settings,whichdefinethenumber
of events that will be consumed before the channel is considered to be filled.
Appropriate values depend greatly on the size of the events and the specific
hardware used, so some experimentation is often required to determine
optimal values for a given environment:
agent_1.channels.channel-1.type= memory
agent_1.channels.channel-1.capacity= 1000
agent_1.channels.channel-1.transactionCapacity= 100
File Channel
The File channel, which persists events to disk, is the most commonly used
Flume channel. To increase the durability of the data placed in the channel,
it persists it to disk for a time. Using a checkpoint and data log system, it
is similar to the system used by Kafka. The exception is that it enforces size
constraints rather than time constraints in the same way as the Memory
channel.
Most File channel definitions can set the
channel.type
property and
perhaps the checkpoint and data file locations as in this example:
agent_1.channels.channel-1.type= file
agent_1.channels.channel-1.checkpointDir= /home/flume/
checkpoint
agent_1.channels.channel-1.dataDirs= /home/flume/data
Like Kafka, the File channel can specify multiple data directories to take
advantage of JBOD configurations. Also, like Kafka, using a RAID0
configuration will probably yield better performance in practice due to the
ability to parallelize writes at the block level.
Java Database Connection (JDBC) Channel
Despite the name, the JDBC channel does not actually allow connections to
arbitrary databases from Flume. A better name for it would be the Derby
channel as it only allows connections to an embedded Derby database. This
channel is the slowest of the three, but it has the most robust recoverability.
