Database Reference
In-Depth Information
agent_1.source.source-1.deserializer=LINE
If needed, custom deserializers can be supplied. Any class that implements
the
EventDeserializer.Builder
interface can be used in place of the
built-in deserializers.
Implementing a Custom Source
It is rarely necessary, given the array of sources built in to Flume, but it is
possible to implement a custom source object. In Flume, there are two types
of sources: a Pollable source and an Event-Driven source. Pollable sources
are executed by Flume in their own thread and queried repeatedly to move
data onto its associated channels. An Event-Driven source is responsible
for maintaining its own thread and placing events on its channels
asynchronously.
In this section, an Event-Driven source connecting Flume to Redis is
implemented to demonstrate the process. Redis is a key-value store
discussed in-depth in Chapter 6, “Storing Streaming Data.” In addition to
providing storage, it also provides an ephemeral publish/subscribe interface
that is used to implement the source.
A custom source always extends the
AbstractSource
class and then
implements either the
Pollable
or the
EventDriveSource
interface.
Most sources also implement the
Configurable
interface to process
configuration parameters. In this case, the Redis source implements both
the
Configurable
and the
EventDrivenSource
interfaces:
package
wiley.streaming.flume;
import
java.nio.charset.Charset;
import
org.apache.flume.Context;
import
org.apache.flume.EventDrivenSource;
import
org.apache.flume.channel.ChannelProcessor;
import
org.apache.flume.conf.Configurable;
import
org.apache.flume.event.EventBuilder;
import
org.apache.flume.source.AbstractSource;
import
redis.clients.jedis.Jedis;
import
redis.clients.jedis.JedisPool;
import
redis.clients.jedis.JedisPoolConfig;
import
redis.clients.jedis.JedisPubSub;
